Deep-Live-Cam + 米家/萤石云摄像头实现本地实时推流换脸

人脸识别 PV:

基座模型

    1. 人脸识别模型：buffalo_l;
    1. 人脸替换模型：inswapper_128_fp16;
    1. 人脸增强模型：GFPGANv1.4

算法介绍

1. 基于inswapper、GFOGAN等模型组成的换脸算法方案，以下简称算法，旨在提供一种便捷的、高效的、实时的人脸替换方案。算法具备以下特点：

  • 只需要提供一张目标图片即可捕获人脸特征，进行替换；

  • 可以针对实时视频流进行处理，或针对媒体文件进行后处理；

  • 可以针对人物的嘴部动作进行实时捕获嘴部掩码，并进行实时替换，实现张嘴动作的同步；

  • 使用生成对抗网络对inswapper模型粗略处理过的面部进行增强，以实现细节上的补充

  • 算法具备一定程度上的环境光抗干扰的能力

2. 算法使用多个模型串联的方式来实现功能流程上的自定义，例如：

  • 基于insightface库arcface算法的人脸识别模型：buffalo_l，其中包括人脸的特征检测模型、性别年龄检测模型、2d、3d关键点检测模型等，主要应用于对源图片和目标图片的人脸检测、特征检测、人脸对齐等；

  • 基于insightface库的人脸交换算法模型：inswapper_128_fp16，具备较高的替换精度和时效性；

  • 基于对抗生成网络的人脸增强模型GFOGANv1.4，应用后对inswapper模型粗换的面部进行细节补全

流程样例

图像

  • 输入换脸图像

  • 输入目标图像

  • 人脸替换输出

  • 人脸增强输出

视频

  • 输入换脸图像

  • 输入目标视频

  • 人脸替换帧输出

  • 人脸增强帧输出

直播流

环境&安装

安装Deep-Live-Cam源码：https://github.com/hacksider/Deep-Live-Cam

安装requirements.txt

windows环境下可直接执行run.py程序进行测试，linux无头界面会无法正确加载ui.py中的功能而报错，将绕过ui.py部分源码，直取功能模块。

基础配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
def parse_args() -> None:
    signal.signal(signal.SIGINT, lambda signal_number, frame: destroy())
    program = argparse.ArgumentParser()
    program.add_argument('-s', '--source', help='select an source image', dest='source_path', default='./source/musk.jpg')
    program.add_argument('-t', '--target', help='select an target image or video', dest='target_path')
    program.add_argument('-o', '--output', help='select output file or directory', dest='output_path')
    program.add_argument('--frame-processor', help='pipeline of frame processors', dest='frame_processor', default=['face_swapper'], choices=['face_swapper', 'face_enhancer'], nargs='+')
    program.add_argument('--keep-fps', help='keep original fps', dest='keep_fps', action='store_true', default=False)
    program.add_argument('--keep-audio', help='keep original audio', dest='keep_audio', action='store_true', default=True)
    program.add_argument('--keep-frames', help='keep temporary frames', dest='keep_frames', action='store_true', default=False)
    program.add_argument('--many-faces', help='process every face', dest='many_faces', action='store_true', default=False)
    program.add_argument('--nsfw-filter', help='filter the NSFW image or video', dest='nsfw_filter', action='store_true', default=False)
    program.add_argument('--map-faces', help='map source target faces', dest='map_faces', action='store_true', default=False)
    program.add_argument('--mouth-mask', help='mask the mouth region', dest='mouth_mask', action='store_true', default=False)
    program.add_argument('--video-encoder', help='adjust output video encoder', dest='video_encoder', default='libx264', choices=['libx264', 'libx265', 'libvpx-vp9'])
    program.add_argument('--video-quality', help='adjust output video quality', dest='video_quality', type=int, default=18, choices=range(52), metavar='[0-51]')
    program.add_argument('-l', '--lang', help='Ui language', default="zh")
    program.add_argument('--live-mirror', help='The live camera display as you see it in the front-facing camera frame', dest='live_mirror', action='store_true', default=False)
    program.add_argument('--live-resizable', help='The live camera frame is resizable', dest='live_resizable', action='store_true', default=False)
    program.add_argument('--max-memory', help='maximum amount of RAM in GB', dest='max_memory', type=int, default=64)# default=suggest_max_memory())
    program.add_argument('--execution-provider', help='execution provider', dest='execution_provider', default=['cuda'], choices=suggest_execution_providers(), nargs='+')
    program.add_argument('--execution-threads', help='number of execution threads', dest='execution_threads', type=int, default=2)# default=suggest_execution_threads())
    program.add_argument('-v', '--version', action='version', version=f'{modules.metadata.name} {modules.metadata.version}')

    # register deprecated args
    program.add_argument('-f', '--face', help=argparse.SUPPRESS, dest='source_path_deprecated')
    program.add_argument('--cpu-cores', help=argparse.SUPPRESS, dest='cpu_cores_deprecated', type=int)
    program.add_argument('--gpu-vendor', help=argparse.SUPPRESS, dest='gpu_vendor_deprecated')
    program.add_argument('--gpu-threads', help=argparse.SUPPRESS, dest='gpu_threads_deprecated', type=int)

    args = program.parse_args()

    modules.globals.source_path = args.source_path
    modules.globals.target_path = args.target_path
    modules.globals.output_path = normalize_output_path(modules.globals.source_path, modules.globals.target_path, args.output_path)
    modules.globals.frame_processors = args.frame_processor
    modules.globals.headless = args.source_path or args.target_path or args.output_path
    modules.globals.keep_fps = args.keep_fps
    modules.globals.keep_audio = args.keep_audio
    modules.globals.keep_frames = args.keep_frames
    modules.globals.many_faces = args.many_faces
    modules.globals.mouth_mask = args.mouth_mask
    modules.globals.nsfw_filter = args.nsfw_filter
    modules.globals.map_faces = args.map_faces
    modules.globals.video_encoder = args.video_encoder
    modules.globals.video_quality = args.video_quality
    modules.globals.live_mirror = args.live_mirror
    modules.globals.live_resizable = args.live_resizable
    modules.globals.max_memory = args.max_memory
    modules.globals.execution_providers = decode_execution_providers(args.execution_provider)
    modules.globals.execution_threads = args.execution_threads
    modules.globals.lang = args.lang

    #for ENHANCER tumbler:
    if 'face_enhancer' in args.frame_processor:
        modules.globals.fp_ui['face_enhancer'] = True
    else:
        modules.globals.fp_ui['face_enhancer'] = False

    # translate deprecated args
    if args.source_path_deprecated:
        print('\033[33mArgument -f and --face are deprecated. Use -s and --source instead.\033[0m')
        modules.globals.source_path = args.source_path_deprecated
        modules.globals.output_path = normalize_output_path(args.source_path_deprecated, modules.globals.target_path, args.output_path)
    if args.cpu_cores_deprecated:
        print('\033[33mArgument --cpu-cores is deprecated. Use --execution-threads instead.\033[0m')
        modules.globals.execution_threads = args.cpu_cores_deprecated
    if args.gpu_vendor_deprecated == 'apple':
        print('\033[33mArgument --gpu-vendor apple is deprecated. Use --execution-provider coreml instead.\033[0m')
        modules.globals.execution_providers = decode_execution_providers(['coreml'])
    if args.gpu_vendor_deprecated == 'nvidia':
        print('\033[33mArgument --gpu-vendor nvidia is deprecated. Use --execution-provider cuda instead.\033[0m')
        modules.globals.execution_providers = decode_execution_providers(['cuda'])
    if args.gpu_vendor_deprecated == 'amd':
        print('\033[33mArgument --gpu-vendor amd is deprecated. Use --execution-provider cuda instead.\033[0m')
        modules.globals.execution_providers = decode_execution_providers(['rocm'])
    if args.gpu_threads_deprecated:
        print('\033[33mArgument --gpu-threads is deprecated. Use --execution-threads instead.\033[0m')
        modules.globals.execution_threads = args.gpu_threads_deprecated

其中，需要注意：

  • max-memory参数在linux系统上过小可能会报错。建议调整为>8G；
  • execution-threads处理的线程数量过大，可能会导致进程不能正确结束；

检查配置

1
2
3
parse_args()
if not pre_check():
    return
1
2
3
for frame_processor in get_frame_processors_modules(modules.globals.frame_processors):
    if not frame_processor.pre_start():
        return

创建帧处理器对象frame_processor，可以为人脸替换器或人脸增强器或两者，并进行初始化检查。具体配置在：

1
program.add_argument('--frame-processor', help='pipeline of frame processors', dest='frame_processor', default=['face_swapper'], choices=['face_swapper', 'face_enhancer'], nargs='+')

资源控制

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
def limit_resources() -> None:
    # prevent tensorflow memory leak
    gpus = tensorflow.config.experimental.list_physical_devices('GPU')
    for gpu in gpus:
        tensorflow.config.experimental.set_memory_growth(gpu, True)
    # limit memory usage
    if modules.globals.max_memory:
        memory = modules.globals.max_memory * 1024 ** 3
        if platform.system().lower() == 'darwin':
            memory = modules.globals.max_memory * 1024 ** 6
        if platform.system().lower() == 'windows':
            import ctypes
            kernel32 = ctypes.windll.kernel32
            kernel32.SetProcessWorkingSetSize(-1, ctypes.c_size_t(memory), ctypes.c_size_t(memory))
        else:
            import resource
            resource.setrlimit(resource.RLIMIT_DATA, (memory, memory))

接入视频

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
cap = cv2.VideoCapture(camera)
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))  # 视频宽度
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))  # 视频高度

frame_processors = get_frame_processors_modules(modules.globals.frame_processors)
source_image = None
prev_time = time.time()
fps_update_interval = 0.5
frame_count = 0
total_count = 0

fps = int(cap.get(cv2.CAP_PROP_FPS))

command = ['ffmpeg',
           '-y',
           '-re',
           '-f', 'rawvideo',
           '-vcodec', 'rawvideo',
           '-pix_fmt', 'bgr24',
           '-s', f"{width}x{height}",
           '-r', str(fps),
           '-i', '-',
           '-c:v', 'libx264',
           '-pix_fmt', 'yuv420p',
           '-preset', 'ultrafast',
           '-f', 'rtsp',
           'rtsp://192.168.9.164:8554/live']
pipe = sp.Popen(command, stdin=sp.PIPE)

图像处理

1
2
3
4
5
6
7
8
9
10
11
12
13
if not modules.globals.map_faces:
    if source_image is None and modules.globals.source_path:
        # 获取人脸
        source_image = get_one_face(cv2.imread(modules.globals.source_path))

    for frame_processor in frame_processors:
        if frame_processor.NAME == "DLC.FACE-ENHANCER":
            if modules.globals.fp_ui["face_enhancer"]:
                # 人脸替换 + 人脸增强
                temp_frame = frame_processor.process_frame(None, temp_frame)
            else:
                # 仅进行人脸替换
                temp_frame = frame_processor.process_frame(source_image, temp_frame)

详细换脸、增强代码见./moduls/processors/frame/face_swapper.py、./moduls/processors/frame/face_enhancer.py

处理视频流

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
while True:
    ret, frame = cap.read()
    total_count += 1
    if total_count % 2 == 0:
        continue
    t = time.time()
    if not ret:
        break

    frame = cv2.resize(frame, (640, 360))

    temp_frame = frame.copy()
    original_height, original_width = frame.shape[:2]

    if modules.globals.live_mirror:
        temp_frame = cv2.flip(temp_frame, 1)

    if not modules.globals.map_faces:
        if source_image is None and modules.globals.source_path:
            source_image = get_one_face(cv2.imread(modules.globals.source_path))

        for frame_processor in frame_processors:
            if frame_processor.NAME == "DLC.FACE-ENHANCER":
                if modules.globals.fp_ui["face_enhancer"]:
                    temp_frame = frame_processor.process_frame(None, temp_frame)
            else:
                temp_frame = frame_processor.process_frame(source_image, temp_frame)
    else:
        modules.globals.target_path = None
        for frame_processor in frame_processors:
            if frame_processor.NAME == "DLC.FACE-ENHANCER":
                if modules.globals.fp_ui["face_enhancer"]:
                    temp_frame = frame_processor.process_frame_v2(temp_frame)
            else:
                temp_frame = frame_processor.process_frame_v2(temp_frame)
    frame_count += 1
    pipe.stdin.write(temp_frame.tobytes())

cap.release()

完整主干代码样例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
import os
import sys
import cv2
import time
import platform
import subprocess as sp

if platform.system() == "Windows":
    from pygrabber.dshow_graph import FilterGraph
from PIL import Image, ImageOps

# single thread doubles cuda performance - needs to be set before torch import
if any(arg.startswith('--execution-provider') for arg in sys.argv):
    os.environ['OMP_NUM_THREADS'] = '1'
# reduce tensorflow log level
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '2'
import warnings
from typing import List, Any
import signal
import shutil
import argparse
import torch
import onnxruntime
import tensorflow
import insightface

import modules.globals
import modules.metadata
from modules.typing import Frame
from modules.utilities import has_image_extension, is_image, is_video, detect_fps, create_video, extract_frames, get_temp_frame_paths, restore_audio, create_temp, move_temp, clean_temp, normalize_output_path
from modules.video_capture import VideoCapturer
from modules.processors.frame.core import get_frame_processors_modules

if 'ROCMExecutionProvider' in modules.globals.execution_providers:
    del torch

FACE_ANALYSER = None
PREVIEW_DEFAULT_WIDTH = 960
PREVIEW_DEFAULT_HEIGHT = 540
warnings.filterwarnings('ignore', category=FutureWarning, module='insightface')
warnings.filterwarnings('ignore', category=UserWarning, module='torchvision')


def parse_args() -> None:
    signal.signal(signal.SIGINT, lambda signal_number, frame: destroy())
    program = argparse.ArgumentParser()
    program.add_argument('-s', '--source', help='select an source image', dest='source_path', default='./source/musk.jpg')
    program.add_argument('-t', '--target', help='select an target image or video', dest='target_path')
    program.add_argument('-o', '--output', help='select output file or directory', dest='output_path')
    program.add_argument('--frame-processor', help='pipeline of frame processors', dest='frame_processor', default=['face_swapper'], choices=['face_swapper', 'face_enhancer'], nargs='+')
    program.add_argument('--keep-fps', help='keep original fps', dest='keep_fps', action='store_true', default=False)
    program.add_argument('--keep-audio', help='keep original audio', dest='keep_audio', action='store_true', default=True)
    program.add_argument('--keep-frames', help='keep temporary frames', dest='keep_frames', action='store_true', default=False)
    program.add_argument('--many-faces', help='process every face', dest='many_faces', action='store_true', default=False)
    program.add_argument('--nsfw-filter', help='filter the NSFW image or video', dest='nsfw_filter', action='store_true', default=False)
    program.add_argument('--map-faces', help='map source target faces', dest='map_faces', action='store_true', default=False)
    program.add_argument('--mouth-mask', help='mask the mouth region', dest='mouth_mask', action='store_true', default=False)
    program.add_argument('--video-encoder', help='adjust output video encoder', dest='video_encoder', default='libx264', choices=['libx264', 'libx265', 'libvpx-vp9'])
    program.add_argument('--video-quality', help='adjust output video quality', dest='video_quality', type=int, default=18, choices=range(52), metavar='[0-51]')
    program.add_argument('-l', '--lang', help='Ui language', default="zh")
    program.add_argument('--live-mirror', help='The live camera display as you see it in the front-facing camera frame', dest='live_mirror', action='store_true', default=False)
    program.add_argument('--live-resizable', help='The live camera frame is resizable', dest='live_resizable', action='store_true', default=False)
    program.add_argument('--max-memory', help='maximum amount of RAM in GB', dest='max_memory', type=int, default=64)# default=suggest_max_memory())
    program.add_argument('--execution-provider', help='execution provider', dest='execution_provider', default=['cuda'], choices=suggest_execution_providers(), nargs='+')
    program.add_argument('--execution-threads', help='number of execution threads', dest='execution_threads', type=int, default=2)# default=suggest_execution_threads())
    program.add_argument('-v', '--version', action='version', version=f'{modules.metadata.name} {modules.metadata.version}')

    # register deprecated args
    program.add_argument('-f', '--face', help=argparse.SUPPRESS, dest='source_path_deprecated')
    program.add_argument('--cpu-cores', help=argparse.SUPPRESS, dest='cpu_cores_deprecated', type=int)
    program.add_argument('--gpu-vendor', help=argparse.SUPPRESS, dest='gpu_vendor_deprecated')
    program.add_argument('--gpu-threads', help=argparse.SUPPRESS, dest='gpu_threads_deprecated', type=int)

    args = program.parse_args()

    modules.globals.source_path = args.source_path
    modules.globals.target_path = args.target_path
    modules.globals.output_path = normalize_output_path(modules.globals.source_path, modules.globals.target_path, args.output_path)
    modules.globals.frame_processors = args.frame_processor
    modules.globals.headless = args.source_path or args.target_path or args.output_path
    modules.globals.keep_fps = args.keep_fps
    modules.globals.keep_audio = args.keep_audio
    modules.globals.keep_frames = args.keep_frames
    modules.globals.many_faces = args.many_faces
    modules.globals.mouth_mask = args.mouth_mask
    modules.globals.nsfw_filter = args.nsfw_filter
    modules.globals.map_faces = args.map_faces
    modules.globals.video_encoder = args.video_encoder
    modules.globals.video_quality = args.video_quality
    modules.globals.live_mirror = args.live_mirror
    modules.globals.live_resizable = args.live_resizable
    modules.globals.max_memory = args.max_memory
    modules.globals.execution_providers = decode_execution_providers(args.execution_provider)
    modules.globals.execution_threads = args.execution_threads
    modules.globals.lang = args.lang

    #for ENHANCER tumbler:
    if 'face_enhancer' in args.frame_processor:
        modules.globals.fp_ui['face_enhancer'] = True
    else:
        modules.globals.fp_ui['face_enhancer'] = False

    # translate deprecated args
    if args.source_path_deprecated:
        print('\033[33mArgument -f and --face are deprecated. Use -s and --source instead.\033[0m')
        modules.globals.source_path = args.source_path_deprecated
        modules.globals.output_path = normalize_output_path(args.source_path_deprecated, modules.globals.target_path, args.output_path)
    if args.cpu_cores_deprecated:
        print('\033[33mArgument --cpu-cores is deprecated. Use --execution-threads instead.\033[0m')
        modules.globals.execution_threads = args.cpu_cores_deprecated
    if args.gpu_vendor_deprecated == 'apple':
        print('\033[33mArgument --gpu-vendor apple is deprecated. Use --execution-provider coreml instead.\033[0m')
        modules.globals.execution_providers = decode_execution_providers(['coreml'])
    if args.gpu_vendor_deprecated == 'nvidia':
        print('\033[33mArgument --gpu-vendor nvidia is deprecated. Use --execution-provider cuda instead.\033[0m')
        modules.globals.execution_providers = decode_execution_providers(['cuda'])
    if args.gpu_vendor_deprecated == 'amd':
        print('\033[33mArgument --gpu-vendor amd is deprecated. Use --execution-provider cuda instead.\033[0m')
        modules.globals.execution_providers = decode_execution_providers(['rocm'])
    if args.gpu_threads_deprecated:
        print('\033[33mArgument --gpu-threads is deprecated. Use --execution-threads instead.\033[0m')
        modules.globals.execution_threads = args.gpu_threads_deprecated


def encode_execution_providers(execution_providers: List[str]) -> List[str]:
    return [execution_provider.replace('ExecutionProvider', '').lower() for execution_provider in execution_providers]


def decode_execution_providers(execution_providers: List[str]) -> List[str]:
    return [provider for provider, encoded_execution_provider in zip(onnxruntime.get_available_providers(), encode_execution_providers(onnxruntime.get_available_providers()))
            if any(execution_provider in encoded_execution_provider for execution_provider in execution_providers)]


def suggest_max_memory() -> int:
    if platform.system().lower() == 'darwin':
        return 4
    return 16


def suggest_execution_providers() -> List[str]:
    return encode_execution_providers(onnxruntime.get_available_providers())


def suggest_execution_threads() -> int:
    if 'DmlExecutionProvider' in modules.globals.execution_providers:
        return 1
    if 'ROCMExecutionProvider' in modules.globals.execution_providers:
        return 1
    return 8


def limit_resources() -> None:
    # prevent tensorflow memory leak
    gpus = tensorflow.config.experimental.list_physical_devices('GPU')
    for gpu in gpus:
        tensorflow.config.experimental.set_memory_growth(gpu, True)
    # limit memory usage
    if modules.globals.max_memory:
        memory = modules.globals.max_memory * 1024 ** 3
        if platform.system().lower() == 'darwin':
            memory = modules.globals.max_memory * 1024 ** 6
        if platform.system().lower() == 'windows':
            import ctypes
            kernel32 = ctypes.windll.kernel32
            kernel32.SetProcessWorkingSetSize(-1, ctypes.c_size_t(memory), ctypes.c_size_t(memory))
        else:
            import resource
            resource.setrlimit(resource.RLIMIT_DATA, (memory, memory))


def release_resources() -> None:
    if 'CUDAExecutionProvider' in modules.globals.execution_providers:
        torch.cuda.empty_cache()


def pre_check() -> bool:
    if sys.version_info < (3, 9):
        update_status('Python version is not supported - please upgrade to 3.9 or higher.')
        return False
    if not shutil.which('ffmpeg'):
        update_status('ffmpeg is not installed.')
        return False
    return True


def update_status(message: str, scope: str = 'DLC.CORE') -> None:
    print(f'[{scope}] {message}')
    # if not modules.globals.headless:
    #     ui.update_status(message)


def destroy(to_quit=True) -> None:
    if modules.globals.target_path:
        clean_temp(modules.globals.target_path)
    if to_quit: quit()


def get_one_face(frame: Frame) -> Any:
    face = get_face_analyser().get(frame)
    try:
        return min(face, key=lambda x: x.bbox[0])
    except ValueError:
        return None


def get_face_analyser() -> Any:
    global FACE_ANALYSER

    if FACE_ANALYSER is None:
        FACE_ANALYSER = insightface.app.FaceAnalysis(name='buffalo_l', providers=modules.globals.execution_providers)
        FACE_ANALYSER.prepare(ctx_id=0, det_size=(640, 640))
    return FACE_ANALYSER


def fit_image_to_size(image, width: int, height: int):
    if width is None and height is None:
        return image
    h, w, _ = image.shape
    ratio_h = 0.0
    ratio_w = 0.0
    if width > height:
        ratio_h = height / h
    else:
        ratio_w = width / w
    ratio = max(ratio_w, ratio_h)
    new_size = (int(ratio * w), int(ratio * h))
    return cv2.resize(image, dsize=new_size)


def get_available_cameras():
    """
    Returns a list of available camera names and indices.
    """
    if platform.system() == "Windows":
        try:
            graph = FilterGraph()
            devices = graph.get_input_devices()

            # Create list of indices and names
            camera_indices = list(range(len(devices)))
            camera_names = devices

            # If no cameras found through DirectShow, try OpenCV fallback
            if not camera_names:
                # Try to open camera with index -1 and 0
                test_indices = [-1, 0]
                working_cameras = []

                for idx in test_indices:
                    cap = cv2.VideoCapture(idx)
                    if cap.isOpened():
                        working_cameras.append(f"Camera {idx}")
                        cap.release()

                if working_cameras:
                    return test_indices[: len(working_cameras)], working_cameras

            # If still no cameras found, return empty lists
            if not camera_names:
                return [], ["No cameras found"]

            return camera_indices, camera_names

        except Exception as e:
            print(f"Error detecting cameras: {str(e)}")
            return [], ["No cameras found"]
    else:
        # Unix-like systems (Linux/Mac) camera detection
        camera_indices = []
        camera_names = []

        if platform.system() == "Darwin":  # macOS specific handling
            # Try to open the default FaceTime camera first
            cap = cv2.VideoCapture(0)
            if cap.isOpened():
                camera_indices.append(0)
                camera_names.append("FaceTime Camera")
                cap.release()

            # On macOS, additional cameras typically use indices 1 and 2
            for i in [1, 2]:
                cap = cv2.VideoCapture(i)
                if cap.isOpened():
                    camera_indices.append(i)
                    camera_names.append(f"Camera {i}")
                    cap.release()
        else:
            # Linux camera detection - test first 10 indices
            for i in range(10):
                cap = cv2.VideoCapture(i)
                if cap.isOpened():
                    camera_indices.append(i)
                    camera_names.append(f"Camera {i}")
                    cap.release()

        if not camera_names:
            return [], ["No cameras found"]

        return camera_indices, camera_names


def create_webcam_preview(camera):
    cap = cv2.VideoCapture(camera)
    width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))  # 视频宽度
    height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))  # 视频高度

    frame_processors = get_frame_processors_modules(modules.globals.frame_processors)
    source_image = None
    prev_time = time.time()
    fps_update_interval = 0.5
    frame_count = 0
    total_count = 0
    fps = int(cap.get(cv2.CAP_PROP_FPS))

    command = ['ffmpeg',
               '-y',
               '-re',
               '-f', 'rawvideo',
               '-vcodec', 'rawvideo',
               '-pix_fmt', 'bgr24',
               '-s', f"{width}x{height}",
               '-r', str(fps),
               '-i', '-',
               '-c:v', 'libx264',
               '-pix_fmt', 'yuv420p',
               '-preset', 'ultrafast',
               '-f', 'rtsp',
               'rtsp://192.168.9.164:8554/live']
    pipe = sp.Popen(command, stdin=sp.PIPE)

    while True:
        ret, frame = cap.read()
        total_count += 1
        if total_count % 2 == 0:
            continue
        t = time.time()
        if not ret:
            break

        frame = cv2.resize(frame, (640, 360))

        temp_frame = frame.copy()
        original_height, original_width = frame.shape[:2]

        if modules.globals.live_mirror:
            temp_frame = cv2.flip(temp_frame, 1)

        if not modules.globals.map_faces:
            if source_image is None and modules.globals.source_path:
                source_image = get_one_face(cv2.imread(modules.globals.source_path))

            for frame_processor in frame_processors:
                if frame_processor.NAME == "DLC.FACE-ENHANCER":
                    if modules.globals.fp_ui["face_enhancer"]:
                        temp_frame = frame_processor.process_frame(None, temp_frame)
                else:
                    temp_frame = frame_processor.process_frame(source_image, temp_frame)
        else:
            modules.globals.target_path = None
            for frame_processor in frame_processors:
                if frame_processor.NAME == "DLC.FACE-ENHANCER":
                    if modules.globals.fp_ui["face_enhancer"]:
                        temp_frame = frame_processor.process_frame_v2(temp_frame)
                else:
                    temp_frame = frame_processor.process_frame_v2(temp_frame)

        current_time = time.time()
        frame_count += 1
        pipe.stdin.write(temp_frame.tobytes())

    cap.release()


def run() -> None:
    parse_args()
    if not pre_check():
        return
    for frame_processor in get_frame_processors_modules(modules.globals.frame_processors):
        if not frame_processor.pre_check():
            return
    limit_resources()

    # 检测可用摄像头
    # available_cameras = get_available_cameras()
    # camera_indices, camera_names = available_cameras
    # print(camera_indices, camera_names)

    # create_webcam_preview(0)
    # create_webcam_preview('./target/webcam.mp4')
    create_webcam_preview('rtsp://USER:PASSWORD@192.168.9.240:8554/njhg_0')


if __name__ == '__main__':
    run()

米家摄像头拉流

搭建miloco

获取资源

1
mkdir /opt/micam && cd /opt/micam

使用docker compose安装，完整安装文件：https://raw.githubusercontent.com/miiot/micam/refs/heads/main/docker-compose.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
services:

  # https://github.com/XiaoMi/xiaomi-miloco
  miloco:
    container_name: miloco
    # image: ghcr.nju.edu.cn/xiaomi/miloco-backend # Official image
    image: ghcr.nju.edu.cn/miiot/miloco:main # Modified image to using HD stream
    network_mode: host
    environment:
      BACKEND_PORT: ${MILOCO_PORT:-8000}
      BACKEND_LOG_LEVEL: ${MILOCO_LOG_LEVEL:-warning}
      TZ: ${TZ:-Asia/Shanghai}
    volumes:
      - ./miloco:/app/miloco_server/.temp
    # NOTICE: Mount configuration files, if you want to use your own configuration files, please mount them here.
    #  - ./miloco/server_config.yaml:/app/config/server_config.yaml
    #  - ./miloco/prompt_config.yaml:/app/config/prompt_config.yaml
    restart: unless-stopped

  # https://github.com/AlexxIT/go2rtc
  go2rtc:
    container_name: go2rtc
    image: ghcr.nju.edu.cn/alexxit/go2rtc
    network_mode: host       # important for WebRTC, HomeKit, UDP cameras
    privileged: true         # only for FFmpeg hardware transcoding
    restart: unless-stopped  # autorestart on fail or config change from WebUI
    environment:
      TZ: ${TZ:-Asia/Shanghai}
    volumes:
      - ./go2rtc:/config     # folder for go2rtc.yaml file (edit from WebUI)

  micam1: &micam
    image: ghcr.nju.edu.cn/miiot/micam:main
    depends_on: [miloco, go2rtc]
    environment: &micam_envs
      MILOCO_BASE_URL: ${MILOCO_BASE_URL:-https://miloco:8000}
      MILOCO_PASSWORD: ${MILOCO_PASSWORD}
      CAMERA_ID: ${CAMERA_ID}
      RTSP_URL: ${RTSP_URL}
      VIDEO_CODEC: ${VIDEO_CODEC:-hevc}
      STREAM_CHANNEL: ${STREAM_CHANNEL:-0}
      TZ: ${TZ:-Asia/Shanghai}
    restart: always
    extra_hosts:
      - "miloco:${MILOCO_HOST_IP:-host-gateway}"

  # More Cameras ...
  micam2:
    <<: *micam
    scale: 0 # Disabled by default, set 1 to enable
    environment:
      <<: *micam_envs
      CAMERA_ID: ${CAMERA_2_ID}
      RTSP_URL: ${CAMERA_2_RTSP_URL}

或使用简约版镜像文件，修改镜像为ghcr.nju.edu.cn/xiaomi/miloco-backend：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
services:

  miloco:
    container_name: miloco
    image: ghcr.nju.edu.cn/xiaomi/miloco-backend
    network_mode: host
    expose:
      - 8000
    environment:
      BACKEND_HOST: 0.0.0.0
      BACKEND_PORT: 8000
      BACKEND_LOG_LEVEL: error
      TZ: Asia/Shanghai
    volumes:
      - /share/Docker/miloco:/app/miloco_server/.temp
      - /share/Camera/logs/milloco:/app/milloco_server/.temp/log
    restart: unless-stopped
1
docker compose up -d

登陆 https://IP:8000

首次登陆会要求输入一个密码，记住并使用MD5工具进行加密，接着登陆小米社区

完成后即可成功进入

参数获取

对首页进行抓包，获取CAMERA_ID

安装go2rtc

使用docker-compose安装：https://hub.docker.com/r/alexxit/go2rtc

1
mkdir /opt/go2rtc && cd /opt/go2rtc
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
services:
  go2rtc:
    container_name: go2rtc
    
    # intel核显使用ghcr.nju.edu.cn/alexxit/go2rtc:master-hardware
    image: ghcr.nju.edu.cn/alexxit/go2rtc:latest
    network_model: host
    priviledge: true
    restart: unless-stopped
    environment:
      - TZ=Asia/Shanghai
      - GID=0
      - UID=0
      - UIDLIST=0
    volumes:
      - "./config:/config"
1
docker compose up -d

登陆 http://IP:1984

进入config目录，进行配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
log:
    level: info
    api: trace
    exec: info
    ngrok: info
    rtsp: warn
    streams: error
    webrtc: fatal

api:
    origin: "*"
    username: "USER"
    password: "PASSWORD"
    
rtsp:
    username: "USER"
    password: "PASSWORD"
    
webrtc:
    username: "USER"
    password: "PASSWORD"
    
streams:
    njhg_0:

完成后点击Save & Restart

搭建micam

如果有多个摄像头，则将micam{n}的数字进行递增。防止冲突新建cam.yml：

1
2
3
4
5
6
7
8
9
10
11
12
13
services:
  micam1:
    image: ghcr.nju.edu.cn/miiot/micam:main
    environment:
      MILOCO_BASE_URL: https://192.168.9.240:8000
      MILOCO_PASSWORD: e10adc3949ba59abbe56e057f20f883e
      CAMERA_ID: 1105929694
      RTSP_URL: # rtsp://{USER}:{PASSWORD}@192.168.9.240:8554/{streams}
      # e.g. RTSP_URL: rtsp://admin:123456@192.168.9.240:8554/njhg_0
      VIDEO_CODEC: hevc
      STREAM_CHANNEL: 0
      TZ: Asia/Shanghai
      restart: unless-stopped

启动

指定cam.yml文件e

1
docker compose -f cam.yml up -d

点击stream即可接入视频流

萤石云摄像头拉流

登录萤石云开放平台：https://open.ys7.com/cn/

找到设备管理器，绑定和找到设备，进入视频直播

找到直播地址

获取rtmp流

Powered by Hexo and Hexo-theme-hiker

Copyright © 2017 - 2026 青域 All Rights Reserved.

UV : | PV :