Recently found an issue with some code running against Amazon’s S3 service. All of a sudden the application, s3cp, was not working as part of our downloader service. The app has been working flawlessly for over a year and suddenly stopped working.
As I tried to debug it I found that the system worked fine locally but not on the server. It was giving an error message around not being able to find one of my accounts buckets. That was odd. First I tried updating the jets3t library used underneath to communicate with S3. The I noticed the server’s Java as bit old but still 1.6. After upgrading both of these still no go.
After taking a break from this I turned on debugging and saw that the authorization interaction with Amazon was returning a 403 error.
To make a long day’s session short for others here is the punch line. Turned out the server’s date was off and Amazon will reject the call if it is too far from the correct time. After setting the servers date and time to the proper value my issue went away.
Sometimes when you debug you need to look at things outside of your code especially if all of a sudden it doesn’t work. In this case the turning point was when the code worked on one machine and not the other. Sadly, we sometimes start upgrading and changing things too much and lose sight of other possibilities.
Of course, if the error message returned by Amazon had more of a specific reason this would have been more quickly solved as well.