It is currently Tue Nov 12, 2019 2:09 pm


Post new topic Reply to topic  [ 11 posts ] 
Author Message
PostPosted: Mon Jun 27, 2005 4:02 pm 
Offline

Joined: Fri May 06, 2005 11:07 am
Posts: 2814
Location: South Lincolnshire
Not long after I started to get back onto observational astronomy I made a point of keeping a paper log of what I do. Nothing grand, just a notebook with written notes, dates, times and the odd rough sketch. When I started doing this I also quickly made a point of making copies of these notes by typing them onto my computer and keeping an online copy on my website.

This electronic copy of my log isn't anything clever; it's done using some elisp code in GNU emacs which generates a HTML template into which I type the text. In addition there's also some code for creating an index and also an index by Messier Object (for the tech types amongst you this is done simply by having subdirectories in the directory structure and using symbolic links in a Unix/Linux filesystem to point to the relevant logs).

However, I quickly became unhappy with this in that I'm mixing form and content in each of the files which means that the data that I've gathered isn't really that easy to search. This got me thinking over the weekend that the sensible thing to do would be to separate the data and the presentation (obvious, I know) and that XML might be the answer.

Now, coming up with an XML schema and some stylesheets for the presentation of the data wouldn't be that hard but I'm worried that in doing so I'd still be missing the point in that I'd probably cobble together my own data format such that the data wouldn't be useful to anyone else (well, I doubt my observations are that useful at the moment, but let's pretend for a moment that they could be at some point in the future). So, I went off looking for projects that have done something similar.

http://www.ecet.vtc.edu/~pchapin/astronomy/ is one example of someone doing just the thing I was thinking about; he also points to another project along those lines http://observation.sourceforge.net/. Further searching on Google didn't turn up a whole lot more.

So, before I either start knocking together my own thing or try working with one of the above, I wanted to ask:
  • Has anyone else looked into this for their own use, where did you get to?
  • Does anyone else do this sort of thing? If so, what do you use?
  • Is anyone aware of any other projects doing this sort of thing?
  • Does anyone know if there has been any sort of effort to standardise the keeping of electronic observing logs?
  • Has anyone in the SPA ever looked into this (perhaps some sort of computing SIG)?
  • Does anyone know of any other groups (especially in the UK) that have had a go at tackling this problem?
  • Is there anyone else out there who has also been thinking about this and fancies working on the problem too?

_________________
Logs | Blog | Weather | WX Blog | Photography (Lincolnshire) | Are you on the map? - I am


Top
 Profile Send private message  
 
 Post subject: Logging observations
PostPosted: Tue Jun 28, 2005 4:37 pm 
Offline

Joined: Tue May 24, 2005 2:19 am
Posts: 171
Location: Northampton
hi DaveP
In the July Ver of Sky At Night, there is an Astro Planner (in the ocver disc) I have had a look at she seems to be ok.

I wounder if this might somehow help.

Alastair

_________________
Engineer, businessman, and Astronomer


Top
 Profile Send private message  
 
 Post subject: Re: Logging observations
PostPosted: Tue Jun 28, 2005 5:06 pm 
Offline

Joined: Fri May 06, 2005 11:07 am
Posts: 2814
Location: South Lincolnshire
Asteeleleith wrote:
In the July Ver of Sky At Night, there is an Astro Planner (in the ocver disc) I have had a look at she seems to be ok.

I have that so I'll have a look -- thanks for the pointer. Although it wasn't so much software I was after it was a (near-)standard method of holding observing logs for easy parsing, passing about, storage and the like. I generally use GNU/Linux for most things so pre-existing software probably doesn't help a whole lot.

Currently I'm playing with an XML format for my own logs that, with the aid of XSLT, can transform the data into the HTML I use right now. That way I get a far more parsable format while retaining my current online format.

_________________
Logs | Blog | Weather | WX Blog | Photography (Lincolnshire) | Are you on the map? - I am


Top
 Profile Send private message  
 
 Post subject:
PostPosted: Wed Jun 29, 2005 9:52 pm 
Offline

Joined: Fri Dec 03, 2004 12:51 pm
Posts: 2451
Location: Stoke-on-Trent
You've raised some interesting issues in this post Dave. Is the attempt here to achieve a standardised format that interested individuals could then implement on their own platforms for personal use, or to come up with an application and repository hosted on the Internet where anyone could submit an observation?

It will be interesting to see if your research turns up any more sites or projects. I put together a small application for personal use a few years back (non-web based), but found that I actually preferred using a hand-written log. Somehow it always seems more personal and tangible, and I can carry it around with me. It's true that it is of no relevance to anyone else, and the observations are of no scientific importance, but I get great enjoyment from picking it up off the bookshelf and looking back through it occasionally. I have to confess I never found the same was true of the electronic version; it lacked the scruffy handwriting, appalling drawings and other subtleties that helped recall very specific things about an observation. Eventually, tired of entering observations into both a manual and electronic log, it fell out of use and I suspect it only resides on a backup CD somewhere in a bottom drawer.

However, if I knew that my contributions were to part of a wider knowledge base, then this might motivate me to use an electronic log. Coming up with a standard template is always going to be tricky. Would it be a generic one, or a more detailed set of templates dependent upon the type of observation being recorded, e.g. data relating to an occultation event would differ from that of a meteor watch or a planetary observation? I'm a great believer in keeping things simple, the more complicated something is, the less likely it is to be used.

I suspect there must be a lot of sites out there that gather data from the wider amateur astronomical community. In a way, perhaps this forum is a good example of that; the presentation is the phpBB bulletin board software, allowing posts to be created in specific topic areas, and the data is presumably held in a back-end SQL database. Organisations gathering occultation timings, variable star estimates etc. must also be good examples. I notice that the Heavens-Above website allows people to record their observations, and to date have over 80,000 observations logged, which does show that people are interested in contributing in this way.

It will be very interesting to see where you take this project though, keep us posted.

Jeff.


Top
 Profile Send private message  
 
 Post subject:
PostPosted: Thu Jun 30, 2005 10:31 am 
Offline

Joined: Fri May 06, 2005 11:07 am
Posts: 2814
Location: South Lincolnshire
jeff.stevens wrote:
Is the attempt here to achieve a standardised format that interested individuals could then implement on their own platforms for personal use, or to come up with an application and repository hosted on the Internet where anyone could submit an observation?

Well, "none of the above" and "all of the above" and some places inbetween. As I said in my initial post, this started because I was making "backup" copies of my logs onto my website but wasn't happy with the "quick hack" approach that I took. The logs are written directly as HTML and, as such, the data is obscured by the markup language. My thinking was (and it's an obvious approach) that it would make more sense to write the logs with some sort of markup that emphasises the meaning of the data and then work from that. Hence the reason that I went looking for XML schemas for doing this sort of thing because translating them to HTML and other formats is easy enough.

So, having decided to do this, I then got wondering about if I should just invent a markup that fits my own style of log keeping or if someone has already worked on this and produced something. That then, of course, got me thinking about if anyone like the SPA or the BAA (I've sent an email to the BAA about this) has looked into this and have come to any conclusions.

So, it was more a case of "I'm going to do something about this anyway but if there's an emerging standard out there it might make sense to use it or a subset of it".

jeff.stevens wrote:
It will be interesting to see if your research turns up any more sites or projects.

I've not found any other projects so far.

jeff.stevens wrote:
I put together a small application for personal use a few years back (non-web based), but found that I actually preferred using a hand-written log.

Actually, the application side of things was the last thing I was thinking about; I was thinking more about the format of the data. I do now have a little project underway and, so far, I'm writing the XML data by hand -- I might look to writing an application for creating/editing logs at some point in the future but that's a secondary concern.

As for keeping logs in a notebook: as I said in my first post, that's what I do now. I'll also carry on doing that. I'm not looking into this with a view to replacing my notebook, it will always be the method via which I make my notes "in the field". While I do generally carry a Pocket PC with me everywhere I go and while I do own a laptop too I wouldn't dream of using them to make notes "in the field" -- for many reasons. There are a few reasons for me keeping an electronic version of my logs and the main one is that I've got a backup copy should anything happen to my paper notebook. On top of this it's easier to index the electronic version (currently I maintain a general index and an index by messier object (the latter being possible evidence of the slowest Messier Marathon ever ;))).

jeff.stevens wrote:
It's true that it is of no relevance to anyone else, and the observations are of no scientific importance, but I get great enjoyment from picking it up off the bookshelf and looking back through it occasionally.

Hmm, I beg to differ. I'd be the first to say that my current observing sessions have no scientific importance at all and I'll take your word in regard to your logs, but that sort of misses the point. Other people's logs serve a useful purpose other than "scientific importance". For example: it's very useful to read other people's logs and see what sort of results they're able to get with what sort of equipment and under what conditions. So, if I'm seeing whatever feature (or lack of) on Jupiter and I'm not sure if I should/shouldn't be able to do it with my setup it's very helpful to read the logs of someone with a similar setup and know if my experience is the norm or not.

jeff.stevens wrote:
However, if I knew that my contributions were to part of a wider knowledge base, then this might motivate me to use an electronic log. Coming up with a standard template is always going to be tricky. Would it be a generic one, or a more detailed set of templates dependent upon the type of observation being recorded, e.g. data relating to an occultation event would differ from that of a meteor watch or a planetary observation? I'm a great believer in keeping things simple, the more complicated something is, the less likely it is to be used.

Well, the two projects I've looked at so far seem to be about making it as detailed as possible although, in one case from what I can tell, it allows for observations to be written in a simple way -- adding more structured detail simply makes the data more useful in terms of parsing and searching.

My own little experiment is about making it as simple as possible. I'm starting to re-write the electronic copies of my logs in XML and have written an XSL file to transform the XML into HTML (in this case transform it back into the HTML form that I initially wrote when I started keeping copies of the logs). You can see an example XML coded log here (that URL is going to be good for around the next 24hrs). This is the XML version of this and the original "source" for that log is here. As you'll see from all of them, the XML version is better structured than the HTML written one and I can get the HTML version back out of the XML (using this (if that URL gives a problem with your browser you might want to do a download from the link instead and then look at the file -- again, the URL's good for at least 24hrs) XSL file in conjunction with Sablotron).

Moreover, once I've got all my logs in XML I can do all sorts of other things with them. Already I've got a more automated system for building the general and Messier indexes (using the index lists that are written into the XML files) and adding searching and the like would be easy enough in the future.

jeff.stevens wrote:
It will be very interesting to see where you take this project though, keep us posted.

Will do. As you can see from the above, I've got a format that works for me right now, is more useful and which allows me to carry on using the framework I've already got available on my website. The nice thing about using the XML approach is that if/when I re-write my website (and I aim to do that in the next year or so, I've had this one since around 2000) I don't have to go back and re-work all the logs to fit the new format, all I need to do is run all the XML files via a different XSL file and the job is done.

I should stress that the point of my own little experiment here isn't to design and propose a format that could be useful for anyone (although anyone wanting to make use of it is more than welcome to do so) and which could form the basis of a shared log site (although, at a push, I guess it could). The point of my original post was to see if anyone else had played with this problem and to see what their solutions looked like. I'm still considering the idea of using one of the existing projects I mentioned in my first post and I'm also hoping for some useful feedback from someone like the BAA (if any feedback from the SPA was available I was hoping that it might be available via this specific forum). The nice thing about what I'm currently playing with is that should a better format become available it should be a lot easier to convert my XML files over to it because I'll already have the electronic copy of my logs in an easy-to-read(-for-the-computer) format.

_________________
Logs | Blog | Weather | WX Blog | Photography (Lincolnshire) | Are you on the map? - I am


Last edited by davep on Tue Jul 05, 2005 9:50 am, edited 1 time in total.

Top
 Profile Send private message  
 
 Post subject: New approach now "live"
PostPosted: Thu Jun 30, 2005 12:11 pm 
Offline

Joined: Fri May 06, 2005 11:07 am
Posts: 2814
Location: South Lincolnshire
I've just finished converting all my logs from HTML to XML and the new pages (which, as intended, look more or less the same as the old ones) are now live: http://www.davep.org/astronomy/logs/

All I've got to do now is starting making interesting observations. ;)

_________________
Logs | Blog | Weather | WX Blog | Photography (Lincolnshire) | Are you on the map? - I am


Top
 Profile Send private message  
 
 Post subject: Index by year
PostPosted: Thu Jun 30, 2005 2:33 pm 
Offline

Joined: Fri May 06, 2005 11:07 am
Posts: 2814
Location: South Lincolnshire
And, because it seems to make sense, I've added an index by year.

_________________
Logs | Blog | Weather | WX Blog | Photography (Lincolnshire) | Are you on the map? - I am


Top
 Profile Send private message  
 
 Post subject:
PostPosted: Thu Dec 29, 2005 10:23 pm 
Offline

Joined: Wed Dec 07, 2005 8:31 pm
Posts: 182
Location: Staffordshire, UK, 52° 48' N 1° 36' W
Dave,

I've worked in IT for 20 years now and one thing I can assure you of is that standardisation is almost impossible to achieve within a single company (let alone a wider user base such as the net) if people are left to their own devices!

Has anyone else noticed that the only consumer item that seems to have a truly universal standard is....the battery! You can buy these anywhere (to my knowledge) without fear of need for adaptors etc.

cheers
Mitch

_________________
Hardware - Opticron MM2 (15-40x52) Spotting Scope + 10x42 Monocular, Skywatcher 200mm Newt
Visit my web log at
http://skywatching-uk.blogspot.com/


Top
 Profile Send private message  
 
 Post subject:
PostPosted: Fri Dec 30, 2005 6:21 am 
Offline

Joined: Fri May 06, 2005 11:07 am
Posts: 2814
Location: South Lincolnshire
Mitch wrote:
I've worked in IT for 20 years now and one thing I can assure you of is that standardisation is almost impossible to achieve within a single company (let alone a wider user base such as the net) if people are left to their own devices!

As someone who has worked in IT for getting on for 17 years I can confirm that I've observed this too. Although, as you'll have noticed, things tend to be a little (not much, but a little) better out in the wilds of the net.

Of course, I wasn't looking for a standard (de facto or de jure) that everyone uses -- I was looking for any efforts to build a standard.

No, I never did find one with any weight behind it.

_________________
Logs | Blog | Weather | WX Blog | Photography (Lincolnshire) | Are you on the map? - I am


Top
 Profile Send private message  
 
 Post subject:
PostPosted: Fri Dec 30, 2005 6:51 am 
Offline

Joined: Thu Sep 22, 2005 2:47 pm
Posts: 91
Dave,
As with most things the writer knows nowt. But, cannot one keep a log in Exel for example, maybe attached to Access for core info?
And No, He's not trying to be clever, He would just like to know.
Maybe extend the writers understanding.
Good new year to you,
Moonstruck.


Top
 Profile Send private message  
 
 Post subject:
PostPosted: Fri Dec 30, 2005 7:02 am 
Offline

Joined: Fri May 06, 2005 11:07 am
Posts: 2814
Location: South Lincolnshire
I can't think of a reason why you can't. Personally I wouldn't (and don't) but there's no obvious technical reason why you couldn't do that.

I've seen a couple of people do this and at least one book I've read (Astronomy Hacks) suggests doing it.

_________________
Logs | Blog | Weather | WX Blog | Photography (Lincolnshire) | Are you on the map? - I am


Top
 Profile Send private message  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 11 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group