Data

July 23, 2025

Weather logging data sets

I am going to just run this script at weekends, so all the data is uploaded as soon as I can after collection. Once transferred to the gopher server, the data is deleted from the Raspberry Pi.

gopher://gopher.vern.cc/1/~zleap

      [42] Weather data June 9th 2025
      [43] Weather data June 10th 2025
      [44] Weather data July 5th 2025
      [45] Weather data July 6th 2025
      [46] Weather data July 12th 2025
      [47] Weather data July 13th 2025

As previously stated, all this is provided 'as is'. I have no intention, at this present time, of sharing by other means.

Tags

#Data,#Science,#Weather,

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

So been experimenting with some of the options for ansiweather, partly as schools need to teach data logging as part of the national curriculum computing, however not all schools have the kit required.

As I have this running on a Raspberry Pi, I thought it may be possible to set one up to log just the location and temperature.

ansiweather -l paignton -p false -h false  -w false -i false >temponly.txt

so the above will display the location and temperature, but exclude pressure, humidity, wind and UV index.

now=$(date +"%m_%d_%Y")
echo "Filename : /nas/backup_$weather.csv"
echo $(date "+ %D, %T"), $(ansiweather -l Paignton -a false) >> weather_$now.csv

Control what is sent to stdout.

now=$(date +"%m_%d_%Y")
echo "Filename : /nas/backup_$weather.csv"
echo $(date "+ %D, %T"), $(ansiweather -l torquay -p false -h false  -w false -i false) >> torquay_$now.csv

Final script

now=$(date +"%m_%d_%Y")
echo "Filename : /nas/backup_$weather.csv"
echo $(date "+ %D, %T"), $(ansiweather -l torquay -p false -h false  -w false -i false) >> torquay_$now.csv

Tags

#Bash,#BashScripting,#Logging,#Data

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

June 9, 2025

Logging the weather

I have modified the script that runs every 10 minutes to record the output from the ansiweather utility, This still saves the output to a file but adds the current date to the file name.

now=$(date +"%m_%d_%Y")
echo "Filename : /nas/backup_$weather.csv"
echo $(date "+ %D, %T"), $(ansiweather -l Paignton -a false) >> weather_$now.csv

So the output looks like

 Humidity: 71% - Pressure: 1023 hPa
06/09/25, 14:20:01, Weather in Paignton: 18 °C - UVI: 7.03 - Wind: 1.79 m/s WSW - Humidity: 69% - Pressure: 1024 hPa
06/09/25, 14:30:01, Weather in Paignton: 18 °C - UVI: 7.03 - Wind: 2.24 m/s SW - Humidity: 70% - Pressure: 1023 hPa
06/09/25, 14:40:01, Weather in Paignton: 18 °C - UVI: 7.03 - Wind: 3.13 m/s W - Humidity: 69% - Pressure: 1023 hPa
06/09/25, 14:50:01, Weather in Paignton: 18 °C - UVI: 7.03 - Wind: 1.79 m/s SW - Humidity: 69% - Pressure: 1023 hPa
06/09/25, 15:00:01, Weather in Paignton: 18 °C - UVI: 7.03 - Wind: 1.79 m/s WSW - Humidity: 71% - Pressure: 1023 hPa
06/09/25, 15:10:01, Weather in Paignton: 18 °C - UVI: 7.03 - Wind: 1.34 m/s SSW - Humidity: 71% - Pressure: 1024 hPa

As this runs on a headless pi4, I can login and copy the data files locally before uploading to my gopherspace (Look it up)

gopher://vern.cc/1/~zleap

Tags

#Bash,#BashScripting,#Data,#Logging,#Gopher,#RaspberryPi, #Weather

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

May 6, 2025

Big Tech and AI Training

Meta, Facebook, and Instagram AI is coming for EU data — Here's what you need to know (and how to opt out)

Opt out of AI data training by big techI

This will happen from 27th May, you need to opt out or your data will be used to train their AI.

Join the Fediverse

Tags

#BigTech,#Meta,#Facebook,#Instagram,#WhatsApp,#GDPR,#Data.#Fediverse,#Mastodon,#PixelFed,#Matrix,#XMPP,#ActivityPub,#Replacement,#Privacy,#Rights,#DataProtection,#EU,#UK

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

April 25, 2025

Bash Scripting 20 – Write date and data a file

So following on from the previous scripts, it may be useful to record both the date and some data associated with that date to a file.

I have found out how to do this, but also tidied my script up, removed some unneeded code (quite a lot) and this should work.

echo "date" "data" # for reference, not written to file 
for ((i = 0 ; i < 10 ; i++)); do
    echo $(date), $((RANDOM % 20)) # echo to screen (stdout)
    echo $(date), $((RANDOM % 20)) > log3.csv
done

Produces

date data
Fri 18 Apr 12:48:16 BST 2025, 0
Fri 18 Apr 12:48:16 BST 2025, 15
Fri 18 Apr 12:48:16 BST 2025, 13
Fri 18 Apr 12:48:16 BST 2025, 6
Fri 18 Apr 12:48:16 BST 2025, 6
Fri 18 Apr 12:48:16 BST 2025, 13
Fri 18 Apr 12:48:16 BST 2025, 18
Fri 18 Apr 12:48:16 BST 2025, 5
Fri 18 Apr 12:48:16 BST 2025, 3
Fri 18 Apr 12:48:16 BST 2025, 14

Which is fine, but we can refine the date output so it is shorter with

date "+DATE: %D TIME: %T"

  date "+%D"   display date no label
  date "+%T"  display tine no label

If we want this in a csv file it may help to add a comma

date "+DATE: %D, TIME: %T"

Remove the labels DATE: & TIME:

date "+ %D, %T"

Produces

date "+ %D, %T"
 04/20/25, 14:58:19

My final entry, in the script run by cron, at least as far as time and date is concerned

echo $(date "+ %D, %T"), $(ansiweather -l Paignton -a false) >> weather.csv

As this sends the output to weather.csv, If we remove the redirect it will go to stout (standard output) we can just run the following on the terminal 'as is'

echo $(date "+ %D, %T"), $(ansiweather -l Paignton -a false)

This you should end up with

04/29/25, 15:44:14, Weather in Paignton: 16 °C - UVI: 5.91 - Wind: 0.89 m/s NNE - Humidity: 75% - Pressure: 1024 hPa

I have just copied and pasted the line in to the terminal and removed the >> weather.csv as that appends the csv file with the data.

As we all have different use cases, this solution is fine for me, I can adjust to suit further. This is far more useful for a CSV file, especially if you want to load the data in to a spreadsheet.

See fourth link below for full details. * Sources of help

https://stackoverflow.com/questions/40175868/how-to-create-csv-file-using-shell-script
https://stackoverflow.com/questions/1194882/how-to-generate-random-number-in-bash
https://linuxhandbook.com/bash-loops/
https://stackoverflow.com/questions/25242626/bash-redirect-output-to-log-file-with-day-date-in-the-name
https://phoenixnap.com/kb/linux-date-command

Tags

#Bash,#Bashscripting,#BashScripting,#Files,#Write,#Data,#Time,#Data

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

April 20, 2025

Bash scripting 18 – Using data files 4

So, carrying on from the previous article(s).

Using the guide at [2]. We can set a few labels up so the graph looks nicer.

Once in gnu plot you can issue commands, then run replot to redraw the graph.

gnuplot> plot 'log.csv'
gnuplot> plot 'log.csv' with lines
gnuplot> set title 'Example Plot'
gnuplot> set xlabel 'x axis'
gnuplot> set ylabel 'y axis' 
gnuplot> replot
gnuplot> set key top right
gnuplot> replot

By running replot we can see the results are what we want before carrying on.

All this is just a small sample of what can be done, I will explore more on this once I have some useful, rather than randomly generated data to plot or do things with.

References

1 gnuplot 2 gnuplot examples

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#BashScripting,#Files,#UsingDataFiles,#Data,#gnuplot,#graphs

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

April 19, 2025

Bash scripting 17 – Using data files 3

So, carrying on from the previous article(s). I am going to be using gnuplot to display the data. I found a useful site, that has examples of how to use the software, from basic to more advanced examples.

Notes: As with bash gnu plot has command history, so you can use arrow keys to move up / down between previously entered commands.

Based on the information at [2] we can produce a very basic plot of what is in log.csv

Load GNUPlot

gnuplot

and enter

plot 'log.csv'

This example isn't all that good really, but it at least produces some output. I have modified one of the examples further down that page

plot 'log.csv' using 1:2 with lines, 'log.csv' using 1:2 with lines

So the following data set

Should produce something like

gnuplot 1

Adding extra columns

We probably need more than two columns. This is easily done by modifying the loop in the script gendata1.csv. It is a good idea to copy this to a new script so it keeps the original as is

cp gendata1.sh gendata2.sh

As gendata1.sh has write permissions, these will be preserved in the new copy of the file. We can now edit gendata2.sh

echo "number" "data" # for reference, not written to file 
for ((i = 0 ; i < 10 ; i++)); do
    echo $i, $((RANDOM % 20)), $((RANDOM % 20)) # echo to screen (stdout)
	echo $i, $((RANDOM % 20)), $((RANDOM % 20)) >> log.csv # write to file
    #write_csv $(($i, $RANDOM % 10))
done

So all we are doing here is adding in, note the comma

, $((RANDOM % 20))

As the only difference between the lines

echo $i, $((RANDOM % 20)), $((RANDOM % 20)) # echo to screen (stdout)
echo $i, $((RANDOM % 20)), $((RANDOM % 20)) >> log.csv # write to file

Is that the first line, writes to the screen, it may be better to test the script out, check it does what you want, then modify the line that writes to a file once you are happy.

References

1 gnuplot 2 gnuplot examples 3. GNU Plot and LaTeX, included for reference

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data,#gnuplot,#graphs,#BashScripting

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

April 18, 2025

Bash scripting 16 – Using data files 2

So, carrying on from the previous article(s). I am going to be using gnuplot and awk, and see what I can do with the data generated by my script.

It would be useful to be able to remove content in the file we are using with awk, so new data is not added to existing data.

We can do this by running:-

echo " " > log.csv

This should ensure the file is empty

we can then rerun

./gencsv1.sh

And check contents with

echo csv.log

So according to the article, we can list the data in the 2nd column

awk -F, '{ print $2}' log.csv

 2
 7
 14
 4
 5
 7
 17
 19
 0
 19

and on a similar note, to list what is in the first column

awk -F, '{ print $1}' log.csv

and we can calculate the sum using

awk -F, '{ sum +=$2} END { print sum }' log.csv
94

and the average

awk -F, '{ sum +=$2} END { print sum/NR }' log.csv
8.54545

Chat

I am on the Devon and Cornwall Linux user group mailing list and also their matrix channel as zleap, it is better to ask there, that way others can answer too.

Tags

#Bash,#Bashscripting,#Files,#UsingDataFiles,#Data,#BashScripting

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

April 17, 2025

Bash scripting 15 – Using data files 1

So, carrying on from the previous article. I am going to be using gnuplot and awk, and see what I can do with the data generated by my script.

So the first task, if of course to install gnu plot

sudo apt install gnuplot

It is also a good idea at this stage to determine version numbers of the tools we are using

gnuplot -V
gnuplot 6.0 patchlevel 0

and

awk -V
GNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.1, GNU MP 6.3.0)
Copyright (C) 1989, 1991-2022 Free Software Foundation.

This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see http://www.gnu.org/licenses/.

April 16, 2025

Bash scripting 14 – Generating data files

Issue 249, May 2025 of Linux Magazine, p52-55, had an article on using awk, and gnu plot to manipulate data. In order to try and make use of this, and learn more, I am trying to figure out how to write a csv file.

Write 10 lines to the first column should be in sequence e.g. 1-10 and the 2nd column should be random numbers. This should be written to a file.

So after some digging I managed to come up with this, by modifying some existing solutions

write_csv(){
    echo \"$1\",\"$2\" > log.csv # write to file, use >> to append
}

#sources of help
echo "number" "data" # for reference, not written to file 
for ((i = 0 ; i < 10 ; i++)); do
    echo $i, $RANDOM # echo to screen (stdout)
	echo $i, $RANDOM >> log.csv # write to file
    #write_csv $(($i, $RANDOM % 10))
done

The screen output for me was

number data
0, 17109
1, 2872
2, 22572
3, 7228
4, 27923
5, 6335
6, 20004
7, 32343
8, 10005
9, 21905

While the file output (log.csv) was similar but without the headings, which I just put in for my own reference, the file does work as a csv as I can open in LibreOffice Calc.

As the numbers are quite big, a closer examination, promoted me to make the following modification

echo $i, $((RANDOM % 20)) # echo to screen (stdout)
echo $i, $((RANDOM % 20)) >> log.csv # write to file

This fixes the 2nd column to produce a number between 1 and 20.

number data
0, 17
1, 12
2, 12
3, 16
4, 2
5, 13
6, 10
7, 6
8, 16
9, 16
cat log.csv
0, 3
1, 17
2, 8
3, 16
4, 10
5, 18
6, 5
7, 14
8, 11
9, 11

The script was called gencsv1.sh, which as expected needed execute permissions

chmod +x gencsv.sh

References

In order to help me I used the following, so giving credit to these sources as would be expected.

Tags

#Bash,#Bashscripting,#Files,#Random,#Data,#CSV,

Mastodon	ShellLabs	Join Mastodon

AI statement : Consent is NOT granted to use the content of this blog for the purposes of AI training or similar activity. Consent CANNOT be assumed, it has to be granted.

Paul Sutton

Data

July 23, 2025

July 18, 2025

June 9, 2025

May 6, 2025

April 25, 2025

April 20, 2025

April 19, 2025

April 18, 2025

April 17, 2025

April 16, 2025