LibreOffice – address label merge (from spreadsheet)

Posted on by rcragun

I’ve been using LibreOffice for over a decade and have only had to do an address label merge twice in that time – both of which have been in the last month. The first time, I actually did a form letter and a label merge; the second time I just had to do an address label merge. In the process, I realized that I totally forgot how to do it after the first time. Whenever that happens, I know that means I need to create a tutorial on here so I can remind myself how to do it if I ever need to do it again. So, here is my tutorial on how to merge labels in LibreOffice. (NOTE: I’m using LibreOffice 7.1.6.2.)

The idea behind a label merge, just like a form letter merge, is that you have a bunch of address information in a spreadsheet or database and, rather than having to enter all of that separately into a document to print labels, you’ll just have the software create the labels from the data you already have. LibreOffice is fully capable of doing this, but it’s not what I would call “easy” or straightforward. And, in fact, the first step will seem unrelated to merging labels, but it is necessary.

Part 1: Register Your Database with the Address Book Data Source Wizard

Before creating any labels, the first thing you need to do is create your database. I’m assuming you have a spreadsheet with addresses, like this one:

Make sure that you have column names at the top as those can become the field names in the database. It’s always a good idea to separate out your fields like I have done: first name, last name, street, city, state, zip. Of course, if you’re outside the US, your columns will be different. Adjust accordingly.

Okay, now we get to the weird part that is going to seem unrelated to merging these addresses into labels. A quick explanation may help. Basically, you need to turn this list of addresses into a database that LibreOffice has registered so it can then pull those fields when it generates the labels. This isn’t hard to do. LibreOffice will walk you right through it with a wizard. But it seems counterintuitive that you have to do this before you can start the mail merge. Just think of this as getting everything set up.

To create your database, click on File -> Wizards -> Address Data Source:

When you click on that, you’ll get this window:

This is Step 1 of 5 for the wizard. It’s basically saying, “Hey, let’s use existing data instead of having to enter new data.” Groovy. That will save us time. Unless you have your address data stored in Firefox or Thunderbird, select “Other external data source.” Then click on “Next” and you’ll get this next screen:

This is Step 2 of 5 for the wizard. On this screen, click on “Settings” and you’ll get a completely separate window:

(Did I mention that this is a bit confusing and complicated. It’s actually really smart, when you think about what the programmers did here, but it isn’t straightforward at all that you have to do all of this first. We’re only about halfway done.)

You have a lot of options you can choose from on this screen. I’m using a Spreadsheet, so I’ll leave that selected. But you can see from this screenshot that you choose from a variety of database sources:

Once you’ve selected the source for your data (again, I’m using a spreadsheet), select “Next” and you’ll get this screen:

Here, you need to choose your data source. Click on “Browse” and go find the spreadsheet you are going to use, like this:

Click on “Open” once you’ve selected it. That will take you back to the previous screen. If you want, you can click on “Test Connection.” Assuming everything worked, you should get this nice but relatively uninformative window:

Click on “Finish” and you’ll go back to the wizard:

This is Step 4 of 5 (note, Step 3 was skipped because I only had one field in my spreadsheet). On this screen, you can click on “Field Assignment,” which will let you match the fields in your imported data to the fields that are common for addresses. (NOTE: This is not required. You can skip this step if you want. It just changes the names of the fields later.) Here’s what the screen should look like:

Just to be clear, the Data source field is what LibreOffice is currently calling the newly imported data (in the image above, it just says “Addresses”). You’ll get a chance to change this later. The Table field (mine is called “Sheet1”) is the table in your spreadsheet where the address information was stored. You can then click on the various fields below where it says “Field Assignment” and align your information with the information options given, like this:

Once you’re done, hit OK and you’ll head back to the wizard.

This is Step 5 of 5. Phew. We’re almost done with this first part. You have a couple of options here. The first is a checkbox that says “Embed this address book definition into the current document.” You can unclick that if you want. Alternatively, you can leave it checked and make it part of the spreadsheet you are working on, which is fine. It really kind of depends on what your future plans are with this data. If this is data that you think you will use repeatedly for mail merges and you’ll probably be updating the data in the future, it might be a good idea to leave that checked. The second option is the one you definitely want to leave checked because we’re going to need it accessible for the label merge we’re going to do next. That option is also a checkbox that says “Make this address book available to all modules in LibreOffice.” Finally, you can name this address book. Call it something you’ll remember. I’m going with “folk artist addresses.” This changes the name that LibreOffice had temporarily assigned it (see the screenshot just above this). Once you’ve made your selections and named your Address Data, click “Finish”:

And once you hit “Finish,” nothing, right? Well, isn’t that weird.

Something did happen, it’s just “behind the scenes.” You have created a database that is now registered with LibreOffice. It’s just not showing it to you. If you want to see it, here’s how. Click on View -> Data Sources:

You’ll then see this:

You can see that I’ve got four Data Sources registered with LibreOffice. One of those is, of course, the one I just created: folk artist addresses. You can actually work with those data sources here if you’d like. Or, if you want to delete one (say, you made one by mistake or for a tutorial and you’ll never need it again), you can right-click on any of the databases and you’ll see this menu:

If you select “Edit Database File,” the LibreOffice database software will start up. If you select “Registered databases,” you’ll see a list of your databases with an option to select them and delete them, like this:

Okay. That does it for the prep work and part 1 of the tutorial. Now, with our address book registered, we can actually do the label merge.

Part 2. Label Merge In LibreOffice

Honestly, the first part is the hardest part. This next part goes pretty quick.

From any document in LibreOffice (it can be a Calc or Writer document; doesn’t matter), click on File -> New -> Labels:

You’ll then see this window:

Some explanation is in order. The big box is where you’re going to construct your addresses. I’ll walk you through that. The little check box at the top that has “Address” next to it is kind of nifty. If you’re a good LibreOffice user, when you first installed the software, you’ll click on Tools -> Options and fill in your User Data:

If you actually did that, you can click that little check box by “Address” and you’ll see your information populate the field:

This is a really slick way to, for instance, create business cards. But that’s not what we’re trying to do. So, uncheck that box and let’s get to merging some labels. First, in the drop down menu under “Database,” select the database we just created (and now Part 1 makes sense):

In the drop down menu under “Table,” select “Sheet1” (it’s the only sheet we had; we could always have named it something else). So, your window should look like this now:

Next, we start working with the drop down menu under “Database field.” In that menu, you should see all the fields you had in your spreadsheet:

Let’s build our address label. Select “firstname” then click on the little arrow pointing left:

That will insert that field into the label box, so you’ll see this:

Now, build the rest of your label, keeping in mind that you’ll want to add spaces and punctuation in between the fields, so it looks like this:

Once you’ve got it set up how you want it, we have a few other things we need to do. At the bottom, you need to choose between “Continuous” or “Sheet.” I’m assuming the Continuous is for printers that can print on labels that are connected. I typically print on individual sheets, so I select that. On the bottom right, find the “Brand” you want and then the “Type.” (Honest Aside: LibreOffice doesn’t always have every option I need for this, but you can usually find something that will work.)

We’re almost done. At the top of that window you’ll see two other tabs: “Format” and “Options.” If you want to customize the dimensions of your label, click on the Format tab. I rarely use this option, but it might be useful if you have a custom label you want to print. There is one thing on the “Option” tab that you probably want to select: “Synchronize contents”:

This will allow you to synchronize any formatting you do to the labels later. Trust me, you’ll want this option like 99% of the time. Once you’re done and have your label selected, click on “New Document” at the bottom.

You’ll then get an entirely new Writer file with labels spread out across the document and your fields highlighted in gray:

What you won’t see, unfortunately, is the actual data in those fields. That will come later. What you’ll want to do now is customize your label formatting. So, if you want to change the font, the font size, make the text bold or italicized, now is the time to do it. And, do it all in the top-left cell. In my document, I increased the font and made everything bold:

Remember when I said to choose “Synchronize contents.” Yeah, now is when you’ll want that. You don’t have to format every individual label. Just format the first one, as I said. And when you’re ready, click on the “Synchronize Labels” button in the floating window that popped up:

When you do, the formatting from the first cell transfers to all the other labels. Way easier than formatting each cell individually.

Now, to print the labels. This part is also, unfortunately, a little confusing. Click File -> Print or just hit the printer icon on the toolbar and you’ll get this message:

“Your document contains address database fields. Do you want to print a form letter?”

What an unfortunate prompt. We’re doing a label merge, so “No.” But don’t choose No! We may not want to print a form letter, but there isn’t a separate prompt for doing a label merge. So choose “Yes.” (Nudge to LibreOffice programmers – fix this.)

When you do, you’ll get this window:

This is a little confusing, but it’s basically that same Data Source window you saw in Part 1, but now we are applying it. You’ll see your Address Data Source on the top left. If you select your sheet (Sheet1 in my screenshot) you’ll see all your addresses. You have a couple of options here. You can choose not to print all of the addresses. This is in the Records option on the bottom left. If you want to test this with a single sheet, for instance, you can choose just the first XX addresses and test this out. On the bottom right, you can choose to save this as a File or send it directly to the Printer. Up to you. If you’re wary and don’t want to waste label sheets, this might be a good option. Or if you need to print these on a different printer, that would be helpful. But if you’re ready to print, select “OK” and you’ll get one more chance to change your mind because you’re finally going to see THE LABEL MERGE! (Woot! Rejoice!):

The screenshot above shows you the final window I get before I actually print the labels. And, assuming everything worked correctly, you’ll finally see the address information merged into the labels. Success!

This is not a simple process but there is, as with all things LibreOffice, a lot of customizability, which is why I prefer LO over Microsoft products.

Anyway, that’s how you do labels merge in LibreOffice. And, bonus, if you ever need to use that same address information in the future, it’s already registered in LibreOffice so you can just do this again. If not, you can delete it as I illustrated above.

 326 total views,  9 views today

R – Finding and Setting Working Directory

Posted on by rcragun

R, like many software programs, likes to have a folder or directory on your computer to operate in. When you start R, depending on how you start it and whether or not you previously saved your session, it’s likely that R will set a default “working directory” or folder where it is going to look for files and folders and where it will default to storing whatever it is you are working (e.g., script files, etc.). Many R beginners don’t know that you can change the working directory AND it is really useful to do so.

There are two ways to change your working directory if you are using R with RStudio – from the console or using the menus in RStudio. I’ll show them both.

To begin with, you can use the following code to get your current working directory:

getwd()

When you enter this into the console or run this from a script file, R will indicate what the current working directory is, like this:

In the screenshot above, you can see that I had set my working directory to a folder called “new analysis” for a project I am working on, but it also shows the entire folder hierarchy to my working directory: “/home/ryan/Dropbox/Ryan/writing/transing god/new analysis”. Knowing what my working directory is, makes it much easier to change to my working directory.

To change your working directory, you can copy the folder hierarchy that R provided and simply change it to the new location. So, for instance, if I wanted to change my working directory to a different project that is stored in a different folder, I can just change the folder hierarchy with the following command:

setwd("/home/ryan/Dropbox/Ryan/writing/futurism studies/data/")

Basically, getting your working directory uses the code “getwd()” with nothing in the parentheses, and setting your working directory uses the code “setwd(“/path/to/folder/”)” with the folder hierarchy included inside quotes inside the parentheses. This is how you get and set your working directory from the console.

If you’re using RStudio to interface with R, you can also set your working directory using the point-and-click menus. Click on Session -> Set Working Directory -> Choose Directory:

Once you click on “Choose Directory,” a window will pop up that will let you select the folder that you want to be your working directory:

Navigate to the folder you want to use as your working directory, select it, and hit “Open” and that will set your working directory. You should also see the corresponding code in the console that matches what we did above.

You can, of course, check to make sure that your working directory has changed by using the code above – “getwd()”. It should now be your new working directory.

Why might you want to change your working directory? The major advantage to doing this is that you no longer have to give the full path length to files you want to import, export, or work on. You can simply give the relative path based on the working directory. For instance, if I want to save a table of data from my environment, if I have set my working directory, I don’t have to specify the entire path to the file when I save it, I can simply save it to my working directory with the name, like this:

write.csv(MyData, file = "MyData.csv")

The code above will save the environment object “MyData” as a CSV file into my working directory and call it “MyData.csv”. If I haven’t set my working directory and want to save the same object, I would have to specify the directory where I want to save it, like this:

write.csv(MyData, file = "/home/ryan/Dropbox/writing/someproject/MyData.csv")

Setting your working directory also makes it easier to import files as you only have to use the name of the file you want to import and not the entire folder hierarchy to the file. With my working directory, I could import a CSV file like this:

dataframe1 <- read.csv("MyData.csv", header=TRUE)

Contrast that code with this code that would be required if I had not set my working directory:

dataframe1 <- read.csv("/home/ryan/Dropbox/writing/someproject/MyData.csv", header=TRUE)

Using a working directory saves time and makes it much easier to import and export files from R.

BONUS:

If you ever need to find your folder hierarchy, this is relatively easy to do on Linux or Windows (no promises on Mac). On Linux, just open a file manager (like Dolphin), navigate to the folder you want, then select the folder hierarchy in your file manager, like this:

You can simply copy and paste that into R. If you want to use the console to do it, you can also navigate to the folder in a console on Linux then use the “pwd” (path to working directory) command to get your current directory:

 480 total views,  8 views today

Launching HandBrake 1.4.0 in Kubuntu 21.04

Posted on by rcragun

The fine folks at HandBrake have updated their software and distribution system to a Flatpack approach. Their Flatpack for Linux is based on Gnome, not KDE, which is fine, but it does mean that the GUI is no longer skinned with my system settings on KDE. Oh well…

The bigger issue is that, with a flatpack install, HandBrake is no longer registered with the KDE system and has to be launched from the command line. That isn’t particularly onerous, but it is annoying when I use KDE’s KRunner to launch most of my software (Alt + F2). There is a solution to this that works just fine: add HandBrake to the Applications in the Application launcher.

To do this, right-click on the Application launcher and select “Edit Applications”:

In the window that comes up, select “New Item” to add HandBrake as an Application:

You’ll get a Window to name the new item:

You can then fill in the following details in the General tab:
Name: HandBrake
Description: HandBrake
Comment: launch Handbrake from Flatpack
Command: flatpack run fr.handbrake.ghb

If you want, you can also download the Handbrake logo and select it by clicking on the little square next to Name and Description:

When you’re done, hit “Save” and HandBrake will be registered with the system as an Application.

Now, when you use KRunner (Alt + F2), HandBrake will pop up as an option:

 524 total views,  4 views today

Linux FFMPEG – Flip Vertical Video to Horizontal

Posted on by rcragun

I was recently shooting some videos on my phone to share with someone else and accidentally flipped the phone before filming a clip. I never film vertically as I know that our playback devices are not designed to playback vertical footage. But I screwed up and got a clip that was vertical. Since I would consider sharing such a clip “bad form,” I needed a quick and easy way to flip the clip. Enter FFMPEG.

This can be done very easily with a single line of code at the console or terminal in Linux, assuming you have FFMPEG installed. Here’s the line of code:

ffmpeg -i input_video.mp4 -vf "transpose=2" output_video.mp4

Here’s what the code does. First, it calls “ffmpeg.” Then you tell it what video to input “-i input_video.mp4.” (The “-i” tells FFMPEG that what follows is the input.) If you’re not in the same directory as the video, you can simply add that information (e.g., /home/user/input_video.mp4). Next, the “-vf “transpose=2″” tells FFMPEG to first “-vf” create a filtergraph (basically, apply a filter) which is “transpose.” The “2” indicates how much to rotate the video – 90 degrees in this case. Finally, pick a name for your file and a file extension. That’s it. That will rotate your video from vertical to horizontal.

Here’s an actual example of code that I used:

ffmpeg -i /home/ryan/Desktop/vertical.mp4 -vf "transpose=2" /home/ryan/Desktop/horizontal.mp4

Sources for this post. FFMPEG’s documentation (here and here) and this post.

 730 total views,  3 views today

Linux – Batch Convert .wav to .flac

Posted on by rcragun

I ran into a weird issue the other day where SoundConverter, a GUI for converting audio files, was generating flac files that my audio player couldn’t see. I’m still not exactly sure what the problem was, but in trying to solve the problem, I went ahead and wrote a command to batch convert a folder of .wav (WAV) files to .flac (FLAC) files using FFMPEG. I figured I’d put it up here for me in the future and in case anyone else finds it useful.

First, navigate in a terminal/console to the folder where the audio files are you want to convert to flac. Then run the following command:

for i in *.wav; do ffmpeg -i "$i" -c:a flac "${i%.*}.flac"; done

Breaking this code down… The first part “for i in *.wav” starts the loop by telling the computer to loop through every file in that folder. The second part tells the computer what to do (“do”) with each of those files: load the ffmpeg software and for each file “$i” convert it to flac “-c:a flac”, renaming the file with the same name as before but with the flac file extension “”${i%.*}.flac””. (See here for what these characters do.) When that is complete, the loop is done.

 1,289 total views,  14 views today

LibreOffice – KDE Integration Package for Skinning/Color Management

Posted on by rcragun

Super simple note for future reference and for anyone else running into this issue. I upgraded my version of LibreOffice in Kubuntu 20.10 to a pre-release version to fix a bug. I was running version 7.1.4.2.

When I purged the version of LibreOffice that shipped with Kubuntu 20.10 (6.X – I don’t remember exactly which version), that also purged the KDE integration package that helps LibreOffice interface with the window manager and makes skinning LibreOffice much easier. As a result, when I would change my Global theme in System Settings, only part of the LibreOffice windows would change to reflect that theme. In particular, the sidebar on the right side of LibreOffice wasn’t changing colors with the rest of the window. This was making it impossible for me to see the names of different styles and also looked really weird.

The solution was easy. In Synaptic, search for the “libreoffice-kde” integration package and install it. Now, when I change my Global Theme, the LibreOffice windows change to reflect that.

In short, if you purge the version of LibreOffice in the repositories in KDE and install a newer version, make sure you also install the libreoffice-kde integration package or your LibreOffice windows will behave strangely.

 687 total views,  2 views today

Linux Server – tar files and directories

Posted on by rcragun

I recently had to move all my websites from one virtual private server (VPS) to another. When I only had a few such websites, I was okay with using SFTP (via Filezilla) to download all of the files and then upload them to the VPS. It took a while but I was okay with that. With about a dozen websites I host on my VPS, that was just not an option. It was time to finally try to figure out how to use tar more effectively on my server.

Why tar? Tar is similar to zip in that it combines lots of files and/or directories into a single file, a tarball or archive, which makes it much easier and way faster to move. Computers can move one large file faster than they can move lots of small files that, combined, make up the same size. When you have to move roughly 8 gigabytes and hundreds of thousands of files, it is far easier to do so by putting all of those files into a single tarball than moving the files individually. That’s why tar is way better for what I was trying to accomplish.

Sidenote: Tar is just a format for packaging all of the files/directories together. Typically, the files are also compressed using something like gzip, leading to a tarball with the extension “tar.gz.” It is possible to just combine the files without compressing them as will be detailed below.

Why was I so reticent to use tar to move my files? Because my prior experience with tar had resulted in several tarbombs, which are a nightmare. Basically, I had unpacked a tarball into the wrong directory, which resulted in thousands of files being in the wrong place, necessitating me having to figure out which files I should keep and which I should get rid of individually. That took more time than I saved by using tar. And since I was doing everything via SSH on a remote VPS, there was no easy way to clean up the mess. Even so, it was time to use tar. So, I bit the bullet and figured out how to do this. This post is my guide on how to carefully use tar but avoid tarbombs, which suck. Note, GNU has an entire manual on tar that goes into much greater detail than my post.

What do all those letters mean?

Since we’re using a console or terminal to package all these files and not a GUI, we have to specify what we want to do with the files using some letters. That’s what the letters do. Not to worry, though – I’m not going to go through all of them. There are literally dozens of options. I’ll keep this simple. The basic structure of the tar command is as follows:

tar [letters to specify what to do] [output tarball] [files or directories to add]

I’ll give some specific examples of tar commands below, but, first, let’s cover what the letters we’ll be using mean.

  • c – “create”: This tells tar to create an archive, in contrast to modifying a tarball. (Note: This letter alone cannot create an archive.)
  • f – “file”: This tells tar to modify files relative to the archive. (Note: Just c and f can work to create a tarball as will be shown below.)
  • v – “verbose”: This tells tar to show the progress of the archiving process by showing which files have been added to the archive.
  • z – “gzip”: This tells tar to compress the files in the tarball using gzip.
  • j – “bzip2”: This tells tar to compress the files in the tarball using bzip2.
  • x – “extract”: This tells tar to extract the files in a tarball.

A couple of important notes here regarding the letters. First, the order of the letters does not matter. Second, letters that do the same things should not be included in the command (e.g., “z” and “j” should not both be used).

Basic Examples:

Creating Tarballs

I put a couple of my papers and some images into a folder to use to demonstrate basic uses of tar. Here’s a screenshot so you can see what we’re working with:

First up, I’ll create a tarball of the entire test folder but with no compression and use that to show a couple of important elements of the process. Here’s the code:

cd /home/ryan/Desktop
tar cf test.tar tar.test.directory/

The code above changes my directory (“cd”) to the parent folder (Desktop). Then it calls the “tar” software and tells it to create an archive called “test.tar” and file inside it (“cf”) the entire test directory “tar.test.directory/”. What this code doesn’t do is compress the files. This can be seen when comparing the size of the folder – 5.4 mb – with the size of the newly created tarball – 5.4 mb.

Quickly, try switching the order of the letters, from “cf” to “fc” and you’ll see that the outcome is the same. Also, if that is all you switch in the code, you’ll notice if you re-run the same command that tar will not warn you that it is going to overwrite the previous tarball, it simply does it.

One more item to note that is actually really important, particularly when thinking about extracting an archive, is the folder structure inside the archive. If I open the archive using Ark, you’ll see that, because I navigated to the parent directory in my terminal before creating the tarball, the folder structure inside the archive is from the folder where I created the tarball (in this case, the Desktop directory).

I’m going to create the same tarball but I’m not going to navigate to the parent folder and instead will tell tar which folder to compress and where to store the new tarball:

tar cf /home/ryan/Desktop/test.tar /home/ryan/Desktop/tar.test.directory/

Functionally, this is the exact same command and archives all the same content. However, look at the directory structure inside the tarball when I open it with Ark:

That tar creates a different directory structure depending on the code you use is important, particularly if you want to avoid tarbombing. Why is this important? When you extract the tarball, the same directory structure that is in the archive will be created. If you don’t know what the directory structure is inside the tarball and extract it, that can result in all sorts of problems, some of which I’ll address at the end of this post.

The lesson here: the folder structure is based on what you enter into the command to create the tarball. So, if you don’t want lots of folders in your tarball, navigate to the folder above it and create it there. Otherwise, you’ll get the same folder structure.

Let’s add two additional letters: “v” for verbose so we can see what is added to the tarball and “z” to compress the files that are being added to the tarball. Here’s the code (remember, the order of the letters doesn’t matter):

tar cfvz test.compressed.tar.gz tar.test.directory/

This command calls the tar software, tells it to create (“c”) a tarball and add files (“f”), show the progress (“v”) and compress the files (“z”), saving the resulting tarball as “test.compressed.tar.gz” and the last piece is what should be put into the tarball. Note the modification to the file name to indicate that the tarball has been compressed – “.gz”. This extension is usually a reflection of the compression format. So, if you go with bzip2, it would be “.bz2” instead of “.gz”. Here’s how it looks in the terminal.

The resulting tarball is only 3.4 mb, illustrating that the contents were compressed as they were added to the archive.

Variations on the above command might include replacing the “z” with “j” to compress using bzip2 or “J” to compress using “xz.” Additionally, appending a “p” to the letters will preserve the permissions of the files and directories that are added to the tarball (though that is done by default, so it isn’t necessary to include it).

Modifying a Tarball

After creating a tarball, it’s possible you may need to change the tarball by either adding files to it or deleting files inside it. Here’s how to do each of those.

To add a file to a tarball, use “r” and, of course, “f”, like this:

tar rf test.tar file-to-add.odt

This command calls the tar software and tells it to append (“r”) a file to the archive (the “f” is necessary to tell the software to make changes). The tarball that is modified is next “test.tar.” And what is to be added is last “file-to-add.odt.”

Assuming the tarball you want to add a file to is compressed, files cannot be added. Attempting this will likely give you the error, “tar: Cannot update compressed archives.” Instead, you would need to extract the archive, make whatever changes you want, then create a new compressed tarball.

It is also possible to delete a file from a tarball. This doesn’t involve a letter but a word, “–delete.” The structure of the command is a bit different. Here’s what your code might look like: 

tar --delete --file=test.tar file-to-delete.odt

This command calls the tar software and tells it you want to delete a file “–delete.” You then have to tell it which tarball and which file, which is done with the “–file=” option, which specifies the tarball and then the file is added at the end.

If you aren’t sure what files you have in your tarball, you can always list those using the “–list” command:

tar --list --file=test.tar

This is particularly helpful if you are looking for a specific file to remove from a tarball as it will also tell you if the file is in a subfolder inside the archive. If so, you would need to modify the code to take that into account:

tar --delete --file=test.tar "folder1/folder2/folder with space/file-to-delete.odt"

Do note that, just like adding files to a compressed tarball, deleting files from a compressed archive isn’t possible.

Extracting a Tarball

Here’s how to extract the files inside a tarball. The basic structure is the same, though, there are some things to consider. First, to extract a tarball, replace the “c” from above when creating it with an “x” which means “extract.” So, with our uncompressed tarball, we would extract it by using the following command:

tar xf test.tar

This will call the tar software, the “x” tells it to “extract” the files (“f”) and “test.tar” is the name of the tarball that is being extracted. Note that this code doesn’t specify where to extract the tarball. That can be added as an option that requires adding “-C”. If it’s left blank, the tarball will be extracted in whichever directory the console/terminal is in. Extracting to a specific directory would like like this:

tar xf test.tar -C /extract/into/this/directory

Of course, if your tarball is compressed and/or you want to see the progress of extracting the tarball, add the corresponding letters. If the tarball is a “tar.gz” you’ll need to add the “z” to decompress the files and “v” to see the progress, like this:

tar xfzv test.tar.gz

Lastly, keep in mind that the folder structure inside the tarball will be replicated when the files are extracted. What does that mean? If you extract a tarball into a folder called “home” but inside the tarball the files you want to extract are stored inside nested folders like “home/ryan/Desktop/archive”, when you extract the tarball, the files will end up in “home/home/ryan/Desktop/archive.” See below for what to do in such a situation.

Rules for Using tar:

Rule #1: Pay attention to the folder or directory structure when creating and extracting a tarball.

Rule #2: You cannot add or delete files inside a compressed tarball, only in an uncompressed tarball.

Bonus:

Let me give a specific situation that will illustrate this for a Linux server (I may be speaking from experience here). Imagine you just downloaded the latest version of WordPress and want to extract it into the following directory: /var/www/example.com/public/. First, check the directory structure of the tarball. In this case, the fine folks who package WordPress archive the files inside a folder called “wordpress.” As a result, when you extract the tarball, it is going to create a folder inside the “public” folder called “wordpress” when, in fact, you want the files to be directly stored into the folder “public.” That’s a problem. How do you move those files? The Linux move command, “mv” can do it, but it’s kind of tricky how:

mv /var/www/example.com/public/wordpress/* .

This tells the OS to move all the files in the wordpress directory (the asterisk “*” does this) up one level (the period “.”). Once you do this, you should then remove the wordpress directory:

rm -R wordpress/

 625 total views,  6 views today

Linux Server – Adding a New Domain and WordPress Site – Linode VPS – Ubuntu 18.04

Posted on by rcragun

I have a VPS server with Linode that I use to host about a dozen different websites. All but one of them run on WordPress. Occasionally, I get a request to add another domain and website to the server. It’s not terribly time consuming to do, but it does require a number of specific steps and I never remember all of them. To help me remember them (and perhaps to help someone else), I’m putting together this tutorial.

Step 1: Purchase the new domain. For this tutorial, I’m going to be adding a domain my brother-in-law requested: flyingyoga.us. He’s a pilot but is getting certified as a yoga instructor and wanted to set up a simple website. I use Google Domains to purchase and manage all my domains. So, step 1, decide on what company you want to use to purchase your domains and purchase your domain.

My domains in Google Domains.

Step 2: Change the DNS settings on the new domain to point to Linode’s nameservers. If using Google domains, click on the domain then click on DNS:

Select “DNS” to change the nameservers.

Under Name servers, select “Use custom name servers” and enter “ns1.linode.com” for the first Name server then add a second and enter “ns2.linode.com.” Hit Save.

Here are the custom name servers in Google Domains.

Step 3: You now need to add the domain and then add domain records to your Linode account. Login to your account and select Domains.

Click on “Create Domain”:

click “Create Domain” at the top right.

Enter the domain and the admin email address. Unless you need to do something special with the Records, select “Insert default records from one of my Linodes.” then select your Linode:

Basic domain creation information.

Assuming you don’t need anything special, the defaults should take care of this step and you’re done.

Step 4: Since I already have about a dozen websites running on the server, I’m not going to go into detail on how to install a LAMP stack – Apache, MySQL, and PHP. There are a number of tutorials for doing so. Instead, my next step is to SSH into my server (obviously replace “user” and the IP address with your own) and create the directories where the files for the new website will be hosted. 

ssh user@192.168.0.1

Whenever I log into my server, I use that as an opportunity to run updates.

sudo apt-get update
sudo apt-get upgrade

Next, navigate to the directory where you store your public-facing web files. On my server, it’s /var/www/

cd /var/www/

In that directory, I’m going to create a new folder for the domain:

mkdir flyingyoga.us

I’m then going to navigate inside that folder and create two additional folders: (1) a “public” folder where the actual files go and (2) a “logs” folder for access and error logs.

cd flyingyoga.us
mkdir logs public

Now, navigate back to the main directory where you store all your website files and change the ownership of the directories:

cd ..
sudo chown -R www-data:www-data flyingyoga.us/

This allows Apache to access the information in the folders and since Apache is effectively the web server, that’s important. Don’t skip this step.

Step 5: Download the latest version of WordPress and untar it into the public folder. Where you download it and untar isn’t actually all that important as we’re going to move it to the public folder shortly. 

sudo wget http://wordpress.org/latest.tar.gz
tar -xvf latest.tar.gz
mv wordpress/* /var/www/flyingyoga.us/public/
rmdir wordpress/
rm latest.tar.gz

Just to clarify the above commands. The first line downloads the latest version of wordpress. The second one unpacks wordpress into a folder called “wordpress.” The third line moves all of the files that were just unpacked into the newly created public folder for the domain. The fourth line deletes the now empty “wordpress” folder and the fifth line deletes the wordpress tar.gz download (nice and clean server).

Step 6: It would be nice if we were done, but we’ve got a ways to go yet. Next up, let’s create a MySQL user and database with a corresponding user. This can be done from the command line as well, but I prefer using phpmyadmin.

You’ll need to look up where to find phpMyAdmin on your server.

Navigate to “User accounts” and scroll down to “Add user account.” Click on that and you’ll get this screen:

Click “add user account” to set up a new database and user.

Obviously, choose an appropriate user name. I typically let phpMyAdmin generate a nice strong password. Just below the “Login Information” box is a box that says “Database for user account.” Check “Create database with same name and grant all privileges.” Don’t check below that where it says “Global privileges – Check all.” That would give this user access to all databases you have on the server. Not a good security choice. Write down or copy the username and password to a text file as you’ll need it later. When you’ve got all that done, scroll down to the bottom and select “Go.” That will create your database, the user, with the password you wrote down (you wrote it down or copied it to a text file, right?). You now have the database WordPress is going to use for your website.

Here’s where you can create a new user and database in PHPMyAdmin

Step 7: Next up is creating the website information for Apache. Back to the SSH shell. Navigate to where the Apache websites are stored on your server:

cd /etc/apache2/sites-available

In there, you should see the configuration files for all the websites on your server. Since I already have sites configured, I typically just copy one of the existing configuration files and then edit it according to the new domain:

cp otherdomain.com.conf flyingyoga.us.conf
cp otherdomain.com-le-ssl-conf flyingyoga.us-le-ssl-conf

Since I’m using SSL on all my domains, I have two configuration files per domain. The above commands copy existing configuration files and create new ones for my new domain. Here’s the contents for the first one: flyingyoga.us.conf:

<Directory /var/www/flyingyoga.us/public>
    Require all granted
</Directory>
<VirtualHost *:80>
        ServerName flyingyoga.us
        ServerAlias www.flyingyoga.us
        ServerAdmin ryantcragun@gmail.com
        DocumentRoot /var/www/flyingyoga.us/public

        ErrorLog /var/www/flyingyoga.us/logs/error.log
        CustomLog /var/www/flyingyoga.us/logs/access.log combined

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.flyingyoga.us [OR]
RewriteCond %{SERVER_NAME} =flyingyoga.us
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

And the contents for the second one – flyingyoga.us-le-ssl-conf

<IfModule mod_ssl.c>
<Directory /var/www/flyingyoga.us/public>
    Require all granted
</Directory>
<VirtualHost *:443>
        ServerName flyingyoga.us
        ServerAlias www.flyingyoga.us
        ServerAdmin ryantcragun@gmail.com
        DocumentRoot /var/www/flyingyoga.us/public

        ErrorLog /var/www/flyingyoga.us/logs/error.log
        CustomLog /var/www/flyingyoga.us/logs/access.log combined

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/ryantcragun.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/ryantcragun.com/privkey.pem
</VirtualHost>
</IfModule>

Once you have these updated, you can then tell Apache to load the site:

sudo a2ensite flyingyoga.us.conf
systemctl reload apache2

The first line tells Apache to enable the site. The second line restarts Apache. NOTE: You don’t have to load the ssl configuration file (i.e., flyingyoga.us-le-ssl-conf).

Step 8: Since I have SSL encryption on all of my websites using LetsEncrypt, there is an extra step. This is always the one I forget. Since I’m adding a domain, I have to use the following commands to add a domain to my existing domains on the single SSL certificate that I use for all of my domains. First, let me find the name of my current certificate:

certbot certificates

That provides me the name of my current certificate as well as a list of all of my other domains. Next, I copy all of the existing domains so I can update the certificate and add the two new ones I need to add. The command to then get a new certificate with the added domains is:

certbot --expand -d existing.com,www.existing.com,flyingyoga.us,www.flyingyoga.com

Assuming everything works, this will expand the existing certificate with the new domain and issue a new SSL certificate with all the domains. (NOTE: no spaces between the domains.)

Step 9: Now you can test your server. I always do this by creating a simple html file with the classic “Hello World” in it and putting that into the public directory for the new website:

<!DOCTYPE html>
<html>
    <head>
        <title>Test Page</title>
    </head>
    <body>
        <p>Hello World!</p>
    </body>
</html>

Save that as “index.html” and put it in the public folder. Now, navigate to the new domain in your browser and, hopefully, you’ll see “Hello World!”

Yeah. Website is working!

If you saw “Hello World!” in your browser, everything is working. It’s always a good idea to check that the https redirect is working as well – so you know that your SSL certificate is good and working. The easiest way to do that is to click on the lock icon in your browser and then check the certificate information.

Step 10: Now, the final step to install WordPress – change the name of the index.html file to something else (e.g. “index.html-test”) then reload the page. You should now see the installation guide for WordPress that will ask for your database name, username, and password:

This is the last step to install WordPress on a new domain.

Enter the database information from Step 6 above. Assuming everything goes according to plan, WordPress will populate the database with the relevant fields and your site will be ready:

Here’s the backend of my new wordpress installation.

 2,447 total views,  3 views today

Long Term Storage of Gmail/Email using Mozilla’s Thunderbird (on Linux)

Posted on by rcragun

I have email going back to 2003. There have been times when I have actually benefited from my email archive. Several times, I have gone back 5 or more years to look for a specific email and my archive has saved me. However, my current approach to backing up my email is, let’s say, a borderline violation of Google’s Terms of Service. Basically, I created a second email account that I use almost exclusively (not exclusively – I use it for a couple of other things) for storing the email from my primary Gmail account. However, Google has recently changed its storage policies for their email accounts, which has made me a little nervous. And, of course, I’m always wary about storing my data with Google or other large software companies.

Since I have already worked out a storage solution for old files that is quite reliable, moving my old email to that storage solution makes sense. (FYI, my storage solution is to run a local file server in my house with a RAID array so I have plenty of space and local backups. Certain folders on the file server are backed up in real-time to an online service so I also have a real-time offsite backup of all my important files. In other words, I have local and offsite redundancy for all my important files.)

I’m also a fan of Open Source Software (OSS) and would prefer an OSS solution to any software needs I have. Enter Mozilla’s Thunderbird email client. I have used Thunderbird for various tasks in the past and like its interface. I was wondering if there was a way to have Mozilla archive my email in a way that I can easily retrieve the email should I need to. Easily might be a bit of a stretch here, but it is retrievable using this solution. And, it’s free, doesn’t violate any terms of service, and works with my existing data backup approach.

So, how does it work? And how did I set it up?

First, install Thunderbird and set up whatever online email account you want to backup. I’m not going to go through those steps as there are plenty of tutorials for both of them. I’m using a Gmail account for this.

Once you have your email account set up in Thunderbird, click on the All Mail folder (assuming it’s a Gmail account) and let Thunderbird take the time it needs to synchronize all of your email locally. With the over one hundred thousand emails I had in my online archive, it took the better part of a day to do all the synchronizing.

I had over 167,000 emails in my online archive account.

Once you’ve got your email synchronized, right-click on “Local Folders” and select “New Folder.” I called my new folder “Archive.”

Technically, you could store whatever emails you want to store in that folder. However, you’ll probably want to create a subfolder in that folder with a memorable name (e.g., “2015 Work”). Trust me, it will be beneficial later. I like to organize things by year. So, I created subfolders under the Archive folder for each year of emails I wanted to back up. You can see that I have a folder in the above image for the year 2003, which is the oldest email I have (that was back when I had a Hotmail address… I feel so dirty admitting that!).

The next step is weird, but it’s the only way I’ve been able to get Thunderbird to play “nice” with Gmail. Open a browser, log in to the Gmail account you’re using, and empty your trash. Trust me, you’ll want to have the trash empty for this next step.

Now, returning to Thunderbird, select the emails you want to transfer to your local folder and drag them over the “Trash” folder in your Gmail account in Thunderbird. This won’t delete them but it will assign them the “Trash” tag in Gmail. Once they have all been moved into the Trash, select them again (CTRL+A) and drag them into the folder you created to store your archived emails. In the screenshot below, I’m moving just a few emails from 2003 to the Trash to test this process:

Once the transfer is complete, click on the Local Folder where you transferred the emails to make sure they are there:

And there are the emails. Just where I want them. This also means that you have a local copy of all your emails in a folder exactly where you want them. At this point, you have technically made a backup of all the email you wanted to backup.

To remove them from your Gmail account, you need to do one additional thing. Go back to the browser where you have your Gmail account open, click on the Trash, and empty the Trash. When you do, the emails will no longer be on Gmail’s server. The only copy is on your local computer.

Now, the next tricky part to this (I didn’t say this was perfectly easy, but it’s pretty easy). Thunderbird doesn’t really store the Local Folder emails in an obvious location. But you can find the location easy enough. Right-click your Local Folder where you are archiving the emails and select “Properties.”

You’ll get this window:

Basically, in the Location box you’ll see the folder’s location. This is telling you where to find your Local Folder where your email is stored. On Linux, make sure that you have “view hidden files” turned on in your file manager (I’m using Dolphin), the location is your home folder, followed by your user folder, then it’s inside the hidden “.thunderbird” folder followed by a randomly generated folder that Thunderbird creates. Inside that folder, look for the “Mail/Local Folders” folder. Or, simply:

/home/ryan/.thunderbird/R$ND0M.default-release/Mail/Local Folders/
I have opened all the relevant folders on my computer in Dolphin see you can see the file structure.

Since I created an additional folder, there are two files in my Archive.sbd folder that contain all the emails I have put into that folder: “2003” and “2003.msf.” You can learn more about the contents of those files here, but, basically, the file with no file extension stores the actual contents of the emails. The .msf file is basically an index of the emails (I was able to open them both in Kate, a text editor, and read them fine). In short, you won’t have a bunch of files in your archive. You’ll have two files – one with the contents of the emails and one that serves as an index of the emails that Thunderbird can read. These are the two files that you’ll want to backup.

I’ll admit, this was the part that scared me. I needed to know that I could just take those two files, move them to wherever I wanted to ultimately store them and then, when needed, move them back into my Thunderbird profile and read them just fine. So, I tested it repeatedly.

Here’s what I did. First, I clicked on my backup folder in Thunderbird to make sure all of the emails were there:

I then went into the Local Folders directory and moved the files to where I want to back them up.

I then clicked on a different folder in Thunderbird and then hit F5 to make Thunderbird refresh the Local Folders. It took a couple of minutes for Thunderbird to refresh the folder, but eventually, it did. Then, I selected the 2003 Archive folder again and the emails were gone:

This is what I expected. The emails are in the “2003.msf” and “2003” files on my backup server. Now for the real test. I copied the two backed up files back to the Archive.sbd folder in my Thunderbird profile, selected the parent folder in Thunderbird, and hit F5 to refresh the folders again. It took a minute for the folder to refresh, but eventually, when I clicked on the 2003 folder and…

The emails are back!

It worked!!!

What this means is that you can put all of the email you want to back up into a folder; that folder is stored in your Thunderbird profile. You can then find the relevant .msf file and its corresponding data file, move them wherever you want for storage and, if needed, move them back to your Thunderbird profile (or, technically, any Thunderbird profile using the same folder structure) and you’ll still have all of your email.

This may not be the most elegant method for backing up your email but it’s free, it’s relatively simple and straightforward, and works reliably. Your email is not in a proprietary format but rather in an open source format that can actually be ready in a simple text editor. Of course, it’s easiest to read it in Thunderbird, but you have the files in a format that is open and secure.

BONUS:

If you don’t think you’re going to need to access your email on a regular basis, you can always compress the files before storing them. Given that most of email is text, you’ll likely realize a savings of close to 50% if space is at a premium (once I moved all of my 2003 email to storage, I compressed it and saw a savings of 60%). This will add a small amount of time to accessing the email as you’ll have to extract it from the compressed format, but it could mean pretty substantial space savings depending on how many emails you’re archiving.

EXTRA BONUS:

This is also a way to move emails between computers. I ended up using this approach to move my email archives to my laptop so I could go through my old email while I’m watching TV at night and delete all the emails I’ll never want in the future (you could of course do that with the email online before you archive it). I’m pretty good about deleting useless emails as I go, but I don’t catch them all. With the .msf file and its accompanying data file, I was able to transport my email archives to whichever computer I wanted and modify the file, then return it to my file server for long term storage.

 3,973 total views,  5 views today

Linux – Bulk UnRAR

Posted on by rcragun

If you’re not familiar with RAR files, they are like ZIP files. RAR is an archive format.

I had a collection of more than 150 RAR files in a single folder I needed to unrar (that is, open and extract from the archive). Doing them one at a time via KDE’s Ark software would work, but it would have taken a long time. Why spend that much time when I could automate the process.

Enter a loop bash script in KDE’s Konsole:

for i in *.rar; do unrar x "$i"; done

Here’s how the command works. The “for i in” part starts the loop (note: you can use any letter here). The “*.rar” component indicates that we want the loop to run through all the RAR files in the directory, regardless of the name of the file. The “do” command tells the loop what command to run. The “unrar x “$i”” component tells the software to use the unrar function which unpacks the archive. The “x” in that command tells the software to use the directory structure inside the RAR archive. The final piece of the loop after the second semi-colon – “done” – indicates what the terminal should do once the loop completes.

It took about 20 minutes to run through all the RAR files, but I used that time to create this tutorial. Doing this by hand would have taken me hours.

 2,783 total views,  5 views today