|
|
|
05-14-2021, 12:34 PM
|
#1
|
Human being with feelings
Join Date: Jan 2021
Location: Paris
Posts: 255
|
Enum all files in all subdirectories
Hi,
For a script, I use io.popen to get the list of all files in a folder and its subfolders.
It works well on my Mac. But I noticed that it doesn't work on Windows...
So, I'm trying to use reaper.EnumerateFiles and reaper.EnumerateSubdirectories.
I can make them work. But how do I get the code to get ALL files in ALL subfolders, and subsubfolders, and subsubsubfolders... etc...
This is my code for now :
Code:
separ = package.config:sub(1,1)
function EnumAllFiles(folder)
reaper.EnumerateFiles(folder,-1) -- Rescan
local i = 0
local filelist = {}
while reaper.EnumerateFiles(folder,i) do
local filename = reaper.EnumerateFiles(folder,i)
if string.sub(string.upper(filename),-4,-1) == '.WAV' then
table.insert(filelist,folder..separ..filename)
end
i = i + 1
end
return filelist
end
function EnumAllSubfolders(folder)
reaper.EnumerateSubdirectories(folder,-1) -- Rescan
local i = 0
local folderlist = {}
while reaper.EnumerateSubdirectories(folder,i) do
local subfolder = reaper.EnumerateSubdirectories(folder,i)
table.insert(folderlist,folder..separ..subfolder)
i = i + 1
end
return folderlist
end
rv, folder = reaper.JS_Dialog_BrowseForFolder('','')
if rv then
local folderlist = EnumAllSubfolders(folder)
local filelist = EnumAllFiles(folder)
-- Print
for i, subfolder in ipairs(folderlist) do reaper.ShowConsoleMsg(subfolder..'\n') end
for i, file in ipairs(filelist) do reaper.ShowConsoleMsg(file..'\n') end
end
Thanks
|
|
|
05-14-2021, 01:29 PM
|
#2
|
Human being with feelings
Join Date: Jan 2021
Location: Paris
Posts: 255
|
Ok, I find a solution with defer()
It may not be the easiest, I don't know.
Code:
separ = package.config:sub(1,1)
function GetAllFiles()
if not folderlist[1] then
return
main()
end
-- Get file list
reaper.EnumerateFiles(folder,-1) -- Rescan
local i = 0
while reaper.EnumerateFiles(folderlist[1],i) do
local filename = reaper.EnumerateFiles(folderlist[1],i)
table.insert(filelist,folderlist[1]..separ..filename)
i = i + 1
end
-- Get subdirectories
reaper.EnumerateSubdirectories(folderlist[1],-1) -- Rescan
i = 0
while reaper.EnumerateSubdirectories(folderlist[1],i) do
table.insert(folderlist,folderlist[1]..separ..reaper.EnumerateSubdirectories(folderlist[1],i))
i = i + 1
end
table.remove(folderlist,1)
reaper.defer(GetAllFiles)
end
function main()
-- Just print
for i, file in ipairs(filelist) do reaper.ShowConsoleMsg(file..'\n') end
end
-- Choose main folder
rv, folder = reaper.JS_Dialog_BrowseForFolder('','')
if rv == 1 then
folderlist = {folder}
filelist = {}
reaper.defer(GetAllFiles)
end
|
|
|
05-14-2021, 01:36 PM
|
#3
|
Human being with feelings
Join Date: May 2017
Location: Leipzig
Posts: 6,630
|
You can try Ultraschall-Api, which has the function GetAllRecursiveFilesAndSubdirectories for this.
I use this in numerous scripts regularly.
https://mespotin.uber.space/Ultrasch...Subdirectories
|
|
|
05-14-2021, 01:56 PM
|
#4
|
Human being with feelings
Join Date: Apr 2013
Location: France
Posts: 9,900
|
Just use recursive functions ! 😄
|
|
|
05-14-2021, 02:00 PM
|
#5
|
Human being with feelings
Join Date: Sep 2008
Location: Sweden
Posts: 7,432
|
Quote:
Originally Posted by X-Raym
Just use recursive functions ! 😄
|
Yeah, this is one of the few cases where recursion actually makes sense.
__________________
// MVHMF
I never always did the right thing, but all I did wasn't wrong...
|
|
|
05-14-2021, 02:09 PM
|
#6
|
Human being with feelings
Join Date: Jan 2021
Location: Paris
Posts: 255
|
How works recursive functions ?
|
|
|
05-14-2021, 02:18 PM
|
#7
|
Human being with feelings
Join Date: Apr 2013
Location: France
Posts: 9,900
|
A function which call itself at some point.
Don't forget to add a return conditions though else it will be infinite loop. :P
|
|
|
05-14-2021, 02:31 PM
|
#8
|
Human being with feelings
Join Date: Jan 2021
Location: Paris
Posts: 255
|
Ah ok !
Something like that :
Code:
separ = package.config:sub(1,1)
function GetAllFiles(folder)
reaper.EnumerateFiles(folder,-1) -- Rescan
local i = 0
while reaper.EnumerateFiles(folder,i) do
table.insert(filelist,folder..separ..reaper.EnumerateFiles(folder,i))
i = i + 1
end
reaper.EnumerateSubdirectories(folder,-1) -- Rescan
local i = 0
while reaper.EnumerateSubdirectories(folder,i) do
GetAllFiles(folder..separ..reaper.EnumerateSubdirectories(folder,i))
i = i + 1
end
end
-- Choose main folder
rv, folder = reaper.JS_Dialog_BrowseForFolder('','')
if rv == 1 then
filelist = {}
GetAllFiles(folder)
-- Print
for i, file in ipairs(filelist) do reaper.ShowConsoleMsg(file..'\n') end
end
|
|
|
05-14-2021, 02:45 PM
|
#9
|
Human being with feelings
Join Date: Apr 2013
Location: France
Posts: 9,900
|
Excatly :P
I didn't run the code but it seems to have the logic.
Filelist could be global or as parameter of the recursive function, and with a final return filelist if it was local
|
|
|
05-14-2021, 02:45 PM
|
#10
|
Human being with feelings
Join Date: May 2017
Location: Leipzig
Posts: 6,630
|
This can blow up, if the folder-depth is too high and end the script with an error.
Don't remember how often a recursive function can be called but there's a stack-limit somewhere.
That's, why I implemented it in my function without recursion, so it always returns all filenames and folders.
|
|
|
05-14-2021, 02:53 PM
|
#11
|
Human being with feelings
Join Date: Apr 2013
Location: France
Posts: 9,900
|
@Meo-Ada Mespotine
It seems there is ways to prevent stack overflow even using recursive function:
https://www.lua.org/pil/6.3.html
(not sure about the integration details though)
But nice to have a full proofed function at disposal for sure!
|
|
|
05-14-2021, 03:03 PM
|
#12
|
Human being with feelings
Join Date: Jan 2021
Location: Paris
Posts: 255
|
Thanks all
|
|
|
05-14-2021, 03:30 PM
|
#13
|
Human being with feelings
Join Date: Jan 2021
Location: Paris
Posts: 255
|
So, if I understand your link correctly, I shouldn't have a stack-limit problem with the following code :
Code:
separ = package.config:sub(1,1)
function GetAllFiles(folderlist)
if type(folderlist) ~= 'table' then return end
local childs = {}
for i, folder in ipairs(folderlist) do
reaper.EnumerateFiles(folder,-1) -- Rescan
local i = 0
while reaper.EnumerateFiles(folder,i) do
local filename = reaper.EnumerateFiles(folder,i)
table.insert(filelist,folder..separ..filename)
i = i + 1
end
reaper.EnumerateSubdirectories(folder,-1) -- Rescan
local i = 0
while reaper.EnumerateSubdirectories(folder,i) do
table.insert(childs, folder..separ..reaper.EnumerateSubdirectories(folder,i))
i = i + 1
end
end
if #childs > 0 then
return GetAllFiles(childs)
end
end
-- Choose main folder
rv, folder = reaper.JS_Dialog_BrowseForFolder('','')
if rv == 1 then
filelist = {}
GetAllFiles({folder})
-- Print
for i, file in ipairs(filelist) do reaper.ShowConsoleMsg(file..'\n') end
end
|
|
|
05-14-2021, 03:45 PM
|
#14
|
Human being with feelings
Join Date: Apr 2013
Location: France
Posts: 9,900
|
Code:
for i, file in ipairs(filelist) do reaper.ShowConsoleMsg(file..'\n') end
This will be huge CPU hog.
Concat the table in one single string to have only one console message call:
Code:
reaper.ShowConsoleMsg(table.concat(filelist, "\n") )
|
|
|
05-14-2021, 03:48 PM
|
#15
|
Human being with feelings
Join Date: Jan 2021
Location: Paris
Posts: 255
|
Yes, you are right, of course.
That part of the code was just for the example.
|
|
|
05-14-2021, 03:52 PM
|
#16
|
Human being with feelings
Join Date: Apr 2013
Location: France
Posts: 9,900
|
Now you need to benchmark against meo-ada mespotine function to see how it goes :P
|
|
|
05-14-2021, 04:09 PM
|
#17
|
Human being with feelings
Join Date: May 2020
Posts: 190
|
Coincidentally I just ran into this problem too, without even seeing your post. Here is a link to the thread I just made with some functions that I use: https://forum.cockos.com/showthread.php?t=253577
|
|
|
05-15-2021, 01:16 AM
|
#18
|
Human being with feelings
Join Date: May 2019
Location: Berlin
Posts: 2,200
|
Quote:
Originally Posted by jkooks
|
Haha, me too, wth! Just yesterday actually. Wrote a simple solution that recusively gets all directory mediafiles (wav,mp3,etc.).
(The second parameter of GetDirMediaFilesRecursive is optional. You can pass it an existing table if you want the function to add files into that)
Code:
function GetDirMediaFilesRecursive(dir, files)
local files = files or {}
local sub_dirs = GetSubDirs(dir)
for _, dir in ipairs(sub_dirs) do
GetDirMediaFilesRecursive(dir, files)
end
GetDirMediaFiles(dir, files)
return files
end
function GetSystemPathSeparator()
return reaper.GetOS():match('Win') and '\\' or '/'
end
function GetSubDirs(dir, sub_dirs)
local sep = GetSystemPathSeparator()
local sub_dirs = sub_dirs or {}
local i = 0
repeat
local sub_dir = reaper.EnumerateSubdirectories(dir, i)
if sub_dir then
sub_dirs[#sub_dirs + 1] = dir .. sep .. sub_dir
end
i = i + 1
until not sub_dir
return sub_dirs
end
function IsMediaFile(file)
local extension = file:match('%.([^\\/]+)$')
if extension and reaper.IsMediaExtension(extension, false) then
return true
end
end
function GetDirMediaFiles(dir, files)
local sep = GetSystemPathSeparator()
local files = files or {}
local i = 0
repeat
local file = reaper.EnumerateFiles(dir, i)
if file and IsMediaFile(file) then
files[#files + 1] = dir .. sep .. file
end
i = i + 1
until not file
return files
end
|
|
|
05-15-2021, 03:36 AM
|
#19
|
Human being with feelings
Join Date: Apr 2013
Location: France
Posts: 9,900
|
@Feedthecat
function GetSystemPathSeparator()
could be done in one line without any function (see rodilab code) :P
|
|
|
05-15-2021, 03:49 AM
|
#20
|
Human being with feelings
Join Date: May 2017
Location: Leipzig
Posts: 6,630
|
package.config can be used for this.
Quote:
package.config
A string describing some compile-time configurations for packages. This string is a sequence of lines:
The first line is the directory separator string. Default is '\' for Windows and '/' for all other systems.
The second line is the character that separates templates in a path. Default is ';'.
The third line is the string that marks the substitution points in a template. Default is '?'.
The fourth line is a string that, in a path in Windows, is replaced by the executable's directory. Default is '!'.
The fifth line is a mark to ignore all text after it when building the luaopen_ function name. Default is '-'.
|
https://www.lua.org/manual/5.3/manua...package.config
|
|
|
05-15-2021, 05:18 AM
|
#21
|
Human being with feelings
Join Date: May 2019
Location: Berlin
Posts: 2,200
|
Quote:
Originally Posted by X-Raym
@Feedthecat
function GetSystemPathSeparator()
could be done in one line without any function (see rodilab code) :P
|
Ah yeah that's cool, good to know
|
|
|
07-19-2021, 09:46 PM
|
#22
|
Human being with feelings
Join Date: Apr 2011
Posts: 3,458
|
I was about to code my own function, and then I thought "wait, let's see what the other guys have come up with!".. Thank you all for your efforts! No need for me to code such a function!
I did a comparison of your functions and here are some results for various folders:
Quote:
Number of files : 61066
jkooks post #17 (single dimension array) function time: 0.788358
FeedTheCat post #18 function time (modified to return all files): 0.908353
Rodilab post #13 function time: 1.422284
Ultraschall function time: 1.694644 (add 9-10ms to load Ultraschall)
|
Quote:
Number of files : 191893
jkooks post #17 (single dimension array) function time: 1.295144
FeedTheCat post #18 function time (modified to return all files): 1.607936
Rodilab post #13 function time: 2.362459
Ultraschall function time: 2.389098 (add 9-10ms to load Ultraschall)
|
Quote:
Number of files : 214772
FeedTheCat post #18 function time (modified to return all files): 0.588189
jkooks post #17 (single dimension array) function time: 0.632195
Rodilab post #13 function time: 1.032876
Ultraschall function time: 1.047048 (add 9-10ms to load Ultraschall)
|
Quote:
Number of files : 2167
jkooks post #17 (single dimension array) function time: 0.013433
FeedTheCat post #18 function time (modified to return all files): 0.014892
Rodilab post #13 function time: 0.068692
Ultraschall function time: 0.082826 (add 9-10ms to load Ultraschall)
|
Results for C:\Windows
Quote:
Number of files : 364279
FeedTheCat post #18 function time (modified to return all files): 12.845067
jkooks post #17 (single dimension array) function time: 14.356715
Rodilab post #13 function time: 18.243644
Ultraschall function time: 20.648948 (add 9-10ms to load Ultraschall)
|
Result for my D:\ drive (comparison of the two fastest, several runs)
Quote:
Number of files : 278805
jkooks post #17 function time: 1.542440
FeedTheCat post #18 function time (modified to return all files): 1.672653
Number of files : 278805
FeedTheCat post #18 function time (modified to return all files): 1.546813
jkooks post #17 function time: 1.648630
Number of files : 278805
jkooks post #17 function time: 1.435645
FeedTheCat post #18 function time (modified to return all files): 1.664412
Number of files : 278805
jkooks post #17 function time: 1.375924
FeedTheCat post #18 function time (modified to return all files): 1.518787
Number of files : 278805
jkooks post #17 function time: 1.304172
FeedTheCat post #18 function time (modified to return all files): 1.501868
Number of files : 278805
jkooks post #17 function time: 1.255934
FeedTheCat post #18 function time (modified to return all files): 1.475037
Number of files : 278805
jkooks post #17 function time: 1.283919
FeedTheCat post #18 function time (modified to return all files): 1.290254
|
jkooks's and FeedTheCat's functions are the fastest. Both could be optimized a bit further by avoiding table.insert and by making them local.
Last edited by amagalma; 07-19-2021 at 10:01 PM.
|
|
|
07-19-2021, 11:14 PM
|
#23
|
Human being with feelings
Join Date: Apr 2011
Posts: 3,458
|
I could not resist
I slightly modified/optimized FeedTheCat's functions and combined them into one:
Code:
local sep = package.config:sub(1,1)
local function GetDirFilesRecursive(dir, files)
-- FeedTheCat function mod by amagalma
local files = files or {}
local sub_dirs = {}
local sub_dirs_cnt = 0
repeat
local sub_dir = reaper.EnumerateSubdirectories(dir, sub_dirs_cnt)
if sub_dir then
sub_dirs_cnt = sub_dirs_cnt + 1
sub_dirs[sub_dirs_cnt] = dir .. sep .. sub_dir
end
until not sub_dir
for dir = 1, sub_dirs_cnt do
GetDirFilesRecursive(sub_dirs[dir], files)
end
local file_cnt = #files
local i = 0
repeat
local file = reaper.EnumerateFiles(dir, i)
if file then
i = i + 1
files[file_cnt + i] = dir .. sep .. file
end
until not file
return files
end
After 100 runs on my D:\ drive these are the results:
Quote:
jkooks (one dimension array ) function time : 1.330141
FeedTheCat (mod by amagalma) function time : 1.274728
|
Last edited by amagalma; 07-20-2021 at 01:22 AM.
|
|
|
07-20-2021, 01:24 AM
|
#25
|
Human being with feelings
Join Date: Apr 2011
Posts: 3,458
|
Hmm.. I didn't encounter any stack overflow messages.. Let me test on whole C:\ drive...
|
|
|
07-20-2021, 01:32 AM
|
#26
|
Human being with feelings
Join Date: Apr 2011
Posts: 3,458
|
Test code:
Code:
local path = "C:\\"
local sep = package.config:sub(1,1)
local filelist = {}
local function GetAllFiles(folderlist)
if type(folderlist) ~= 'table' then return end
local childs = {}
for i, folder in ipairs(folderlist) do
reaper.EnumerateFiles(folder,-1) -- Rescan
local i = 0
while reaper.EnumerateFiles(folder,i) do
local filename = reaper.EnumerateFiles(folder,i)
table.insert(filelist,folder..sep..filename)
i = i + 1
end
reaper.EnumerateSubdirectories(folder,-1) -- Rescan
local i = 0
while reaper.EnumerateSubdirectories(folder,i) do
table.insert(childs, folder..sep..reaper.EnumerateSubdirectories(folder,i))
i = i + 1
end
end
if #childs > 0 then
return GetAllFiles(childs)
end
end
local function GetDirFilesRecursive(dir, files)
-- FeedTheCat function mod by amagalma
local files = files or {}
local sub_dirs = {}
local sub_dirs_cnt = 0
repeat
local sub_dir = reaper.EnumerateSubdirectories(dir, sub_dirs_cnt)
if sub_dir then
sub_dirs_cnt = sub_dirs_cnt + 1
sub_dirs[sub_dirs_cnt] = dir .. sep .. sub_dir
end
until not sub_dir
for dir = 1, sub_dirs_cnt do
GetDirFilesRecursive(sub_dirs[dir], files)
end
local file_cnt = #files
local i = 0
repeat
local file = reaper.EnumerateFiles(dir, i)
if file then
i = i + 1
files[file_cnt + i] = dir .. sep .. file
end
until not file
return files
end
collectgarbage("collect")
local mem1 = collectgarbage("count")
local start = reaper.time_precise()
GetAllFiles({path})
time1 = reaper.time_precise() - start
local mem2 = collectgarbage("count")
files1 = #filelist
memusage1 = mem2-mem1
filelist, mem1, mem2 = nil
collectgarbage("collect")
local mem1 = collectgarbage("count")
local start = reaper.time_precise()
local filelist = GetDirFilesRecursive(path)
time2 = reaper.time_precise() - start
local mem2 = collectgarbage("count")
files2 = #filelist
memusage2 = mem2-mem1
Results:
Quote:
files1: 1495305
files2: 1495301
memusage1: 296208 KB
memusage2: 312447 KB
time1: 47.789556700001 secs
time2: 23.751817299999 secs
|
The "1" values are of your function and the "2" values are of my mod.
|
|
|
07-20-2021, 01:41 AM
|
#27
|
Human being with feelings
Join Date: Apr 2011
Posts: 3,458
|
I ran it again without measuring memory usage:
Quote:
files1: 1495357
files2: 1495357
time1: 39.220415100001
time2: 24.718277600001
|
And once again:
Quote:
files1: 1495380
files2: 1495382
time1: 37.8845683
time2: 24.682484199999
|
And once again with mem usage:
Quote:
files1: 1495395
files2: 1495391
memusage1: 296249.16308594
memusage2: 312462.63671875
time1: 39.534345199998
time2: 25.065250899999
|
What is strange is that both functions return each time different numbers of files...
But this doesn't happen with smaller folders...
Edit: Now, that I think of it, it is to be expected when counting all files of C:\.. The number of files changes all the time because of temporary files created by running processes etc..
Bottom line is that the modded function uses only 5% more memory with no stack overflows in real usage (tested as you saw with 1.5million files) and it is about 35% faster.
Last edited by amagalma; 07-20-2021 at 01:56 AM.
|
|
|
Thread Tools |
|
Display Modes |
Linear Mode
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
All times are GMT -7. The time now is 02:22 PM.
|