Republicans argue the pipeline, which would ship oil from Canada and northern states to Texas, would create jobs and improve energy security at a time of surging gasoline prices.
Obama put TransCanada's $7 billion project on hold earlier this year pending further environmental review. He took the unusual step of calling some senators personally ahead of the vote, asking them to reject the proposal.
"He understood that a majority of the American public, a majority at least in the Senate, are strongly in favor of this project," said Senator Richard Lugar, the top Republican on the Senate Foreign Relations committee, who sponsored the bill to take control of the pipeline decision away from Obama.
The Republicans tried to advance their plan as an amendment to a highway funding bill. It failed on a vote of 56-42, four short of the 60 needed to pass, although 11 Democratic senators voted with the Republicans.
Republicans are using the proposal to highlight Obama's delay of the project ahead of November presidential and congressional elections, linking his decision to rising gasoline prices.
"We're going to continue this fight," said Republican Senator John Hoeven of North Dakota, who championed the bill.
He told reporters he hoped the measure might still be attached to the highway funding package when the Senate and House of Representatives work on a final version.
"With gas prices going up every day, with what's going on in the Middle East, I'll tell you what: the pressure is just going to increase on the administration to get this project done," Hoeven said.
Obama has supported construction of the southern leg of the pipeline, and his administration will assess a new route around an environmentally sensitive area of Nebraska once it has been identified, said White House spokesman Clark Stevens.
"Once again, Republicans are trying to play politics with a pipeline project whose route has yet to be proposed," Stevens said. The entire project will take more than two years to build once permits are granted.
GREEN GROUP: 'TEMPORARY VICTORY'
The Keystone amendment was among 30 measures - many of them energy-related - being voted on as the Senate pushes in coming days to renew funding for highways and other infrastructure projects, slated to run out at the end of March.
Earlier, the Senate defeated proposals to expand the area available for offshore oil drilling and extend the time for manufacturers to phase in new pollution regulations set by the Environmental Protection Agency for industrial boilers.
But the Keystone amendment attracted the most attention. The pipeline would carry crude from Canadian oil sands to Texas refineries and would also pick up U.S. crude from North Dakota and Montana along the way.
Environmental groups have fought the project, staging large protests last year that pressured the Obama administration to block approval.
"Today's vote was a temporary victory and there's no guarantee that it holds for the long run," Bill McKibben, founder of 350.org, said in a statement.
"We're grateful to the administration for denying the permit and for Senate leadership for holding the line."
With a 34-64 vote, senators also defeated a proposal from Democratic Senator Ron Wyden that would have blocked exports of oil from the pipeline, as well as refined products made from that oil.
Wyden said lawmakers need to carefully think through projects that would increase exports of oil, fuel and natural gas, lest the exports end up boosting prices for Americans.
"This is just a step in what is clearly going to be an extensive debate," Wyden told Reuters after the vote.
Democratic senators who voted for the Republican Keystone plan included Max Baucus and Jon Tester of Montana, Kent Conrad of North Dakota, Bob Casey of Pennsylvania, Claire McCaskill of Missouri and Jim Webb of Virginia.
Two Republican senators were absent, and all the 45 who were present voted for the amendment.
(Additional reporting by Thomas Ferraro; editing by Mohammad Zargham and Todd Eastham)
