FTP: how to get total number of files in directory
apple Verified ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Hello Folks,
I prefer to use FTP from the command line but cannot figure out how--after having connected to a remote directory via ftp--to list the total number of files in that directory.
In Unix, I would do this:
% ls | wc -l
But hopefully there is also some way in FTP to do the same--but darned if I know how! I do see that FTP clients such as Transmit have this listing, but maybe that's a feature of the software and not ftp.
Thanks in advance for any suggestions/tips.
Doug
15 replies
Sign in or join Beachball Help to reply.
- 0
Jun T. ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Try "pls" instead of "ls" in ftp:
% export PAGER='less -M'
% ftp ftp.remote.host
ftp> plsThis will list the contents of the remote current directory by using the "less" command. Hit "G" to go to the end of the list. Then prompt of less will show how many lines are there. (Type ctrl-G if the prompt shows no info).
Or you can use a local temporary file:
ftp> ls . tmpfile
output to local-file: tmpfile [anpqy?]? y
229 Entering Extended Passive Mode (|||54412|)
150 Here comes the directory listing.
86851 50.55 KB/s
226 Directory send OK.
ftp> !wc tmpfile
1335 12015 85516 tmpfileHere, "ls . tmpfile" indicates to save the listing of the current remote directory "." to the local file "tmpfile", and "!wc tmpfile" is to run "wc tmpfile" locally.
- 0
Bruce.Ferjulian ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
For times like this you may want consider using ( ssh ) and ( scp ).
Your local command [ ls | wc -l ] would translate to [ ssh user@host ls | wc -l ],
this way you don't even need ( ftp ). Once you get the remote list of files,
you can use ( scp ) to get them.-Bruce-
- 0
Bruce.Ferjulian ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Sorry about the strike-though. I didn't know my brackets would be treated as formatting delimiters.
So...
AS-LOCAL : ls -l | wc -l
AS-REMOTE: ssh username@hostname ls -l | wc -l - 0
Nils C. Anderson ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
sorry if someone already suggested this.
curl -l -s -u username
ftp://hostname/directory/ | wc -lMessage was edited by: Nils C. Anderson
- 0
Hawaiian Scuba Dude ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
I've done a lot of ftp scripts 10 years ago, mostly because I could not guarantee my scripted telnet would be available everywhere. The best answer today though is listed throughout this thread and that is to use the ssh family of commands that include ssh for secure remote shell, scp for secure remote copy and sftp which is the ftp interface to scp.
My point is neither telnet or ftp are considered safe or secure anymore because listening to communications is popular, it is referenced in most "wizard" magazines, and usernames and passwords are not encrypted.
Set up your environment first with ssh-keygen. Version 2 is more secure than version 1 if both are still offered. Then use "ssh user@hostname ls | wc -l" to get your number of files. It is simple execution with one command and returns a status.
If you are really looking for a solution using what you are familar with and want to ignore advice concerning security, here is a sample batch script (untested). You can change the filenames, usernames, and hostnames as desired.
rm dir.out 2> /dev/null
echo "user sampleuser samplepw" > script.ftp
echo "dir sampledirectory dir.out" >> script.ftp
echo "quit" >> script.ftp
ftp -n sample.hostname < script.ftp
if [ ! -r dir.out ] # file not created
then
echo login failed
elif [ ! -s dir.out ] # file created but empty
then
echo directory is empty
else
nf=`wc -l dir.out | cut -f1`
echo "There are $nf files in the directory"
fi - 0
Hawaiian Scuba Dude ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Ouch! The http:// is a translation error by the forum application and should be square brackets surrounding the test condition.
- 0
Doug Niven1 ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Hello All,
These are all great tips, thanks.
However, I guess I'm inevitably going to run into the problem I'm trying to overcome: the ~2000 file listing limit of FTP.
I have directories with 3000 or 4000 files and I'm trying to get the exact number in the parent directory, but I just don't think there's a way to do this, other than zip archive the directory and download, which is what I wanted to avoid.
Further brilliant ideas?
Doug
- 0
Cole Tierney ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
{quote:title=Doug Niven1 wrote:}
I have directories with 3000 or 4000 files and I'm trying to get the exact number in the parent directory, but I just don't think there's a way to do this, other than zip archive the directory and download, which is what I wanted to avoid.Further brilliant ideas?
{quote}Did you see Bruce's suggestion to run
ls | wc -l on the remote machine via ssh? When running a remote command via ssh the results are printed to your console and can be saved locally. I think that would solve your problem.--
ColeMessage was edited by: Cole Tierney
- 0
Doug Niven1 ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Yeah, I did see Bruce's note.
But if you read my original posting, I already know how to do that via SSH.
I do not know how to do via FTP, and ls | wc -l doesn't work in FTP.
- 0
Cole Tierney ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Is there a reason you need to use ftp? Just to be clear, if you run:
ssh username@hostname ls -l | wc -l > somefile
...somefile will be on your local machine and not on the remote host as it would if you were logged in.
--
Cole - 0
Doug Niven1 ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Because I have ONLY an FTP account with this system.
SSH and any every other method is NOT available.
This is very common for webservers, FYI.
- 0
Kurt Sakaeda ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
Dear Doug,
If it was me, I would set up a .netrc file like this:
$echo machine isp.com login userID password secretPassword>.netrc
$chmod 400 .netrc
Then I would type a command like this:
$echo dir|ftp [email protected]|grep rw|wc
If ftp sees you have set up a .netrc file it will take the password from the .netrc file for ftp login. This means that ftp does not ask for password & you just bypass security. I am not certain if this works for sftp. But if you do use sftp there is a way to bypass security for sftp too. Just let me know & I will look up my notes.
The echo command feeds the request for directory information to ftp. It is about the same as typing dir at the ftp prompt.
The ftp command runs and sends its output to standard out.
The grep command filters out all of the ftp status junk & just lists the files.
The wc command counts up the number of lines or files.
If I can be of any further help, please feel free to post any further questions.
Your humble unix geek,
Kurt
- 0
Nils C. Anderson ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
the curl example that I gave above uses the ftp account that I have on the system that I tested it on.
- 0
Cole Tierney ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
{quote:title=Doug Niven1 wrote:}
Because I have ONLY an FTP account with this system.
{quote}Thanks for that clarification. If Nils' suggestion doesn't pan out for some reason and you run out of things to try, you might try a cgi shell script:
<pre style="padding-left: .75ex; padding-top: .25em; padding-bottom: .25em; margin-top: .5em; margin-bottom: .5em; margin-left: 1ex; max-width: 80ex; overflow: auto; font-size: 10px; font-family: Monaco, 'Courier New', Courier, monospace; color: #222; background: #eee; line-height: normal">#!/bin/shecho Content-type: text/plain
echols ${QUERY_STRING} | wc -l
</pre>
If you do try this, you'd probably want to password protect it or do some input validation.--
Cole - 0
Jun T. ·
Imported from the Apple Support Forums via the Wayback Machine 🔗
If the ftp server is configured to limit the size of the directory listing to 2000, and if you can access the server only via ftp, then I guess the only way out is to divide 3000-4000 files into subdirectories.