Papa John's founder John Schnatter is "not going quietly," according to his lawyer Patricia Glaser.
And he didn't want to work with singer Kanye West, he said in a letter sent to the board of directors Saturday.
Schnatter doubled down on claims he made during a television interview Friday in which he said media consultant Laundry Service tried to blackmail the pizza chain for $6 million to keep quiet about his use of the N-word during a May conference call.
He sent the letter in advance of an emergency board meeting Sunday, defending himself and criticizing the board for forcing his resignation as chairman before conducting a full investigation into a Forbes article published Wednesday that detailed his use of the racial slur.
“The board asked me to step down as chairman without apparently doing any investigation," Schnatter said in the letter obtained by CNBC. "I agreed, though today I believe it was a mistake to do so. I have checked with corporate governance experts who tell me that this was not a proper action by the Board."
Laundry Service and Papa John's didn't immediately respond to CNBC's requests for comment.
Schnatter resigned his post last week at the behest of Papa John's board after confirming the Forbes' report. However, he said the comments were taken out of context and that he was provoked into using the N-word after Laundry Service executives on the call suggested the pizza chain bring on performer Kanye West as a co-spokesman for television spots and promotions.
Schnatter said he refused to work with West because "he uses the 'N' word in his lyrics," according to the letter. It was later on the call that Schnatter said he used the actual word when pressed whether or not he was a racist.
"I then said something on the order of, Colonel Sanders used the word 'N,' (I actually used the word,) that I would never use that word and Papa John’s doesn’t use that word," Schnatter told the board. "Let me be very clear: I never used the 'N' word in that meeting as a racial epithet, nor would I ever."
Glaser, Schnatter's lawyer, warned the board in a separate letter sent Sunday against removing him altogether after several directors questioned whether he should give up his seat, she said. Glaser told CNBC the board doesn't have authority to remove him without shareholder approval.
That would be a tough vote to win since Schnatter owns almost 30 percent of the outstanding shares, according to FactSet. Glaser asked the board to open an investigation into the call and subsequent events.
"Those individuals were acting on rumor and innuendo, without any investigation — let alone a third-party investigation of the facts," Schnatter wrote of the board's actions.
Papa John's announced late Sunday that Schnatter was prohibited from speaking to the press, was being removed from the company's advertising materials and that a special committee of independent directors revoked his office space at its Louisville, Kentucky, headquarters.
Schnatter reiterated his side of events in the letter, stating that Laundry Service was brought in to conduct a "diversity media training" session ahead of a conference he was attending.
He said Papa John's fired Laundry Service the day after the call took place, but still owed the media agency about $1.3 million. Schnatter said the agency requested $6 million "because they claimed some of their people had been offended by what I had said," he wrote.
One their attorneys threatened "a smear campaign" if Papa John's didn't pay up, Schnatter said, adding that the company ultimately paid Laundry Service $2.5 million.
"I will not allow either my good name or the good name of the company I founded and love to be unfairly tainted,” Schnatter wrote.
WATCH: Papa John's founder claims he was blackmailed
