Project Gutenberg has tons of public domain books for downloading directly to your Kindle.
MobileGuide is a book you can download to your Kindle that has links to many public domain books formatted for the Kindle.
ManyBooks.net, more public domain books.
Internet Archive of guess what? Public domain books! Did’ja see that one coming?
RetroRead will take free public domain ebooks from Google Books, make the format Kindle-Friendly, and then let you download them to your Kindle, all for free!
Go to the Kindle store on Amazon.com (make sure your Kindle is set to United States under ‘Manage You Kindle’), and when searching for books, sort by price, low to high.