Automatizing access to web pages

However it looks like that, particulary since the WWW is gone commercial after its birth among us physicists, web sites are not developed in an optimal way to make that possible or easy.
Compare the following cases ...

• Once I wanted to be able to run every night a script which interrogates a web site at MIT, which keeps the data taken by the All Sky Monitor instrument on board the NASA X-ray astronomy satellite Rossi XTE. The script should monitor the intensity of a few selected X-ray sources in the sky, warn me by e-mail when they undergo some changes, so that I can apply for Target Of Opportunity observation with BeppoSAX, another X-ray astronomy satellite of the Italian Space Agency.
  I could set up this quite easily (see here) with a shell script based on lynx, which required minimal maintenance in case of unforeseen format changes in the site at MIT

• Another time I wanted to be able to run every night a script which interrogates a site at the ESA (European Space Agency) VILSPA station about the schedules of the observations with the ESA XMM-Newton X-ray astronomy satellite. It should e-mail me when some of our targets show up in the schedule, when the observation has being done, and also feed some information into a database I mantain.
  I could set up this too quite easily (see here) with a group of shell scripts based on lynx and mysql, which required some maintenance in case of unforeseen format changes in the site at VILSPA

• A third time I want to be able to interrogate a (project private) database with astronomical data mantained at an institute in Marseille, to extract the optical magnitudes of all objects (and a sky image of the field) within a given distance from each of the X-ray sources observed by us. The result should then be fed into a local database
  This is also arranged quite easily as a set of scripts based on lynx and mysql, which are run manually once our X-ray data have been analysed. They required little maintenance, although some time was spent writing the initial scripts in order to access the Marseille database in an efficient way, bypassing the 3 or 4 tiers of web pages which are normally used for manual access.

  do the above sound exotic to you ? perhaps, although that's part of my normal work. However doing those action automatically was quite easy to do and to mantain. Now consider the following more familiar things ...

• Assume you want to travel somewhere you have never been, and you want to know what the weather (temperature) will be like there. What I want to do is to interrogate some web site with the weather information once per day, and save the temperature and weather into a table for some weeks before I travel.
  I did this several times for various places, at distance of a few months, using a web site at Yahoo, and, as usual, a lynx based script. However I encountered the following difficulties
  • the few interesting textual information is contained in a plethoric file full of unintersting graphics and advertisement, which requires a very painful parsing
  • the format of those files keeps continuously changing from one time to the next (with more and more rubbish being added each time), so I have to rewrite the parsing software each time
  • even the way different localities are indexed into web pages keep changing from time to time

• Assume you want to interrogate periodically the records of your bank account, or of your savings' asset management at your bank, and just print the relevant information in a reformatted way (let's say you want to see the bonds sorted by expiry date) or to store the results to a table, or to add the totals to a list producing a graph of amount vs time.
  First of all I could not do it via lynx because the bank site uses https, and because the site is designed to work with IE and Netscape 7 alone.
  Furthermore the site uses frames, so I cannot even use netscape via remote control because the protocol does not allow to access an individual frame, and the login procedure does not allow to bookmark a dynamic frame in a frameset. So I can only write a script which directs me to click in the wished frame to proceed.
  And then I have to parse the output, which I have found to be both quite unstable in format, but also full of arbitrary patterns which are not easily parsable