Pena, who reached on a bunt as a pinch-hitter in the ninth, helped Atlanta finish off a rally in a game it trailed 4-0 after two innings and 4-2 entering the ninth.
Eric O’Flaherty (2-0) pitched a 1-2-3 ninth for the win, while Craig Kimbrel retired the heart of Washington’s order for his fifth save in five chances.
With one out in the 10th, Stammen (2-1) walked Dan Uggla. After a flyout, Pena hit his first homer of the season.
The Braves, a major league-leading 9-1, handed Washington its first loss in seven home games.
Atlanta tied the game in the ninth, when Washington third baseman Ryan Zimmerman’s throwing error contributed to reliever Drew Storen blowing a save. It was Storen’s first save chance since giving away a ninth-inning lead against the St. Louis Cardinals in Game 5 of an NL division series last season.
On Friday, Washington went up 4-0 after two innings against Atlanta right-hander Julio Teheran.
Bryce Harper hit a two-run, opposite-field shot in the first for his fifth homer, and Denard Span and Jayson Werth tacked on RBI singles in the second.
That advantage stood up for most of the evening, in large part because Ross Detwiler produced his second strong start of the season, allowing one run and four hits in seven innings.
Atlanta’s only run off Detwiler came on Chris Johnson’s solo homer in the seventh, a drive over the 402-foot sign in straightaway center.
But then the Nationals’ bullpen took over. One problem: Manager Davey Johnson wanted to stay away from closer Rafael Soriano, who had pitched each of the previous three days, earning three saves during a sweep of the Chicago White Sox.
And so this one was a stomach-churner for Nationals fans in the crowd of 33,130.
Washington was ahead 4-1 heading to the eighth, with top setup man Tyler Clippard on for Detwiler. And Clippard got two quick outs before everything started to unravel.
Clippard went walk, single, walk, walk — with that last pass to B.J. Upton forcing in a run to make it 4-2. That was it for Clippard, and on came Storen. He got the third out of the eighth on a fly ball to the warning track in center by 26-year-old rookie Evan Gattis.
Then, in the ninth, consecutive singles off Storen put two runners on, and a sacrifice bunt pushed them to second and third. With two outs, Storen walked Jason Heyward to load the bases for Justin Upton, B.J.’s brother.
Justin Upton hit a hard shot to Zimmerman, who tried to throw to second for a forceout that would have ended the game. But Zimmerman’s throw was wide of the bag and sailed into right field. Justin Upton was credited with a single and one RBI, while the second run that scored on the play was unearned because of Zimmerman’s throwing error. That made it 4-all.
Much, much earlier, Teheran gave up four runs and six hits in six innings, while Detwiler turned in a five-strikeout showing.
Span got things started in the first by reaching on a tapper to third baseman Johnson, whose throw pulled Gattis slightly off the bag at first. Johnson initially was charged with a throwing error, but the official scorer later changed his ruling to a hit for Span.
An out later, Harper took a 93 mph outside pitch and sent it barely over the wall in left field for a 2-0 lead.
Span was involved again in the second inning. This time, Teheran walked No. 8 hitter Kurt Suzuki, who moved to second on Detwiler’s sacrifice. Span delivered an RBI single and took second on a throw home that never had a chance to get Suzuki. Span sped home on Werth’s hit to make it 4-0.
NOTES: In the sixth, umpires reviewed Justin Upton’s hit off the top of the wall to see if it was a homer, but ruled it was a double. Upton stayed at second because Detwiler struck out Gattis. ... Gattis played first base for the first time since Class A in 2011. He mostly looked fine, with one bad-looking play in the fifth, when Harper lofted a high popup in foul territory that Gattis lingered under but misjudged, the ball dropping to the ground. Harper eventually walked but was stranded at second. ... RHP Luis Ayala pitched a hitless seventh in his Braves debut. ... Zimmerman played in his 1,000th game with the Nationals.