Gov. David Paterson is formally backing off his plans to bring the Legislature back to Albany on Nov. 15 for a lame duck special session, saying he agrees with Assembly Speaker Sheldon Silver that the Senate chaos requires that date to be pushed back at least several weeks.

We caught up with Paterson at the Capitol in San Juan last night, and he didn’t take kindly to the suggestion that the legislation he says must be addressed prior to the end of the calendar year could just as easily be taken up before the end of the fiscal year (in other words, next April).

“We have to come back before Jan. 1. (Silver is) absolutely right about Nov. 15 because some of those races are still being resolved,” Paterson said.

“I’ll probably move that date back a couple of weeks, and I’ll do it as soon as Monday,” the governor continued. “However we do have to come back to implement federal legislation for education, an OTB plan for New York City and a couple of other things that have to be done, and we will be coming back.”

“We’re going to do it before the end of the fiscal year, and it comes down to who you’re going to believe. And I’m telling you, we’ll be back before the end of the year.”

As you’ll recall, this isn’t the first time the governor has capitulated when the speaker balked at bringing his members back to the Capitol.

Paterson had threatened to call a pre-election session in hopes of getting the Assembly to take up the property tax cap the Senate passed prior to departing last summer. Silver, who is no fan of the cap, said he would be more than happy to discuss it – with the next governor.

On election night, Silver told CapCon’s Jimmy Vielkind: “I think we’ll work out an appropriate time to come back and an appropriate agenda. It will be sometime this calendar year, but without the Senate settled, Nov. 15 seems optimistic.”

And then on Thursday night, during an interview with me at Somos, Silver took it a little further, saying:

“I will work with Governor Paterson until a new governor is the governor of the state. that’s my responsibility as a legislator that’s every member of the Legislature’s responsibility….If he wants to restore vetoed items, he never should have vetoed them to begin with.”

Silver said he hasn’t talked to Cuomo about whether the governor-elect wants the Legislature to return prior to Jan. 1 to address the $315 million hole that has opened in this year’s budget, or if he’d prefer to simply handle it himself in his first executive budget proposal, which will come during the first month of his tenure.

Pressed on whether he will be back at some point this year, period, Silver said:

“I can’t tell you that right now. I think if there’s unfinished business before the end of the year certainly we will do what we have to do.”