Commit Graph

893 Commits

Author SHA1 Message Date
Daniel Pacak
46fb43bc25 chore: Bump up Trivy adapter to v0.4.0
Allows configuring SCANNER_TRIVY_GITHUB_TOKEN environment variable,
which is passed to trivy executable binary when it starts scanning
a given artifact.

This is to increase GitHub requests rate limit from 60 per hours
(for anonymous requests) to 5000 when Trivy download its
vulnerabilities database.

Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-03-16 09:53:16 +01:00
Wenkai Yin
a4a1913598 Repair the count usage during the upgrading
As the count quota is against artifact rather than tag in 2.0, the count usage should be recalculated

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-13 13:59:48 +08:00
Ted Guan
4ac31c6d46
Add API for query supported event types and notify types; Return policy name in last trigger info; Remove project_id unique constraint in table notification_policy (#11029)
Signed-off-by: guanxiatao <guanxiatao@corp.netease.com>
2020-03-11 18:06:58 +08:00
Wenkai Yin(尹文开)
8452100148
Merge pull request #10942 from ywk253100/200305_reference
Persistent the URLs and annotations of artifact references in database
2020-03-11 16:20:18 +08:00
Wang Yan
bd7940217a
upgrade golang version to v1.13.8 (#11006)
The vesrion contains two security bug fix - CVE-2020-0601, CVE-2020-7919

More details, see the golang milestone:

https://github.com/golang/go/issues?q=milestone%3AGo1.13.8+label%3ACherryPickApproved

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-11 12:20:06 +08:00
Ziming Zhang
695a2559be feat(cicd) use unified version as tag name, clean more
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-03-09 17:13:28 +08:00
Ziming Zhang
200c352c35 feat(cicd) use unified version as tag name
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-03-09 15:30:03 +08:00
Daniel Jiang
6d89553c4d
Merge pull request #10937 from reasonerjt/csrf-2.0
Update CSRF mechanism
2020-03-09 12:31:08 +08:00
Wenkai Yin(尹文开)
75eb7a8c5a
Merge pull request #10955 from wy65701436/migrate-access
add sql for migrating access log
2020-03-09 10:00:08 +08:00
Daniel Jiang
ae5ffce83a Update CSRF mechanism
This commit replaces beego's CSRF mechanism with gorilla's csrf library.
The criteria for requests to skip the csrf check remain the same.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-03-09 01:15:54 +08:00
wang yan
288c7790d0 add sql for migrating access log
1, loop each access log, change to resource/resource_type, and insert into audit log
2, loop each first push operation, change it to create repository and insert into audit log.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-06 12:06:12 +08:00
wang yan
2b0b7576b2 Fix gc issue on clean the artifact trash
1, enable dao test for artifact trash
2, set default flush trash table to false
3, hanlder empty parameter in API call
4, add registry auth info into jobservice container

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-06 03:11:31 +08:00
stonezdj(Daojun Zhang)
49619e1907
Merge pull request #10939 from wy65701436/access-log-mgr
add audit logs API
2020-03-05 16:24:21 +08:00
wang yan
df237a5b17 add audit logs API
1, add API entry for get audit logs
2. add audit log manager to hanlder CRUD

Use the new format of audit log to cover differernt resource, artifact/tag/repostory/project

Signed-off-by: wang yan <wangyan@vmware.com>
2020-03-05 11:40:51 +08:00
Wenkai Yin
76c04b0219 Persistent the URLs and annotations of artifact references in database
Persistent the URLs and annotations of artifact references in database

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-03-05 10:54:45 +08:00
Will Sun
a5d9a3b65d
Merge pull request #10863 from AllForNothing/api-center
Fix Api cennter
2020-03-05 10:00:15 +08:00
Wenkai Yin(尹文开)
4fa4c4e74c
Merge pull request #10815 from cd1989/redis-idle-timeout
Set redis idle timeout for core
2020-03-03 14:10:22 +08:00
Wenkai Yin
4c9b59c904 Migrate artifact data in 2.0
Abstract extra attributes and annotations for artifacts stored in database

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-28 18:09:02 +08:00
jwangyangls
572510c82d
Merge pull request #10755 from Snaacker/master
Fix typo
2020-02-28 11:59:51 +08:00
Daniel Jiang
1823c984f7
Merge branch 'master' into redis-idle-timeout 2020-02-27 22:01:22 +08:00
AllForNothing
d41c5496a2 Fix Api cennter
Signed-off-by: AllForNothing <sshijun@vmware.com>
2020-02-27 15:55:20 +08:00
Wenkai Yin
02c2647e1e Use the repository name of artifact model
As we store the repository name in the artifact table, we can use it direclty in the code to reduce the database query

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-26 13:37:09 +08:00
stonezdj(Daojun Zhang)
a7e5873f46
Merge pull request #10821 from stonezdj/20200224_remove_notification
Remove registry notification and change core health check url
2020-02-25 13:34:37 +08:00
Ziming Zhang
94230b5e19 feat(cicd) fix some build problem
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-02-25 12:05:39 +08:00
stonezdj
6005101c95 Remove registry notification and change /api/ping
Update config.yaml.jinja to remove notification
Change api/ping in core/Dockerfile

Signed-off-by: stonezdj <stonezdj@gmail.com>
2020-02-25 11:24:21 +08:00
Wang Yan
948d45604c Revise the GC job flow,
1, set harbor to readonly
2, select the candidate artifacts from Harbor DB.
3, call registry API(--delete-untagged=false) to delete manifest bases on the results of #2
4, clean keys of redis DB of registry, clean artifact trash and untagged from DB.
5, roll back readonly.

Signed-off-by: wang yan <wangyan@vmware.com>
2020-02-24 18:29:55 +08:00
Wenkai Yin
bd204464f3 Remove dead code
Remove dead code

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-23 17:11:46 +08:00
dechen
e642a73280 Set redis idle timeout for core
Signed-off-by: dechen <xxyydream@gmail.com>
2020-02-23 12:31:56 +08:00
Wenkai Yin
9312b788dc Upgrade the artifact table
Split the table artifact into artifact and tags, and populate related data

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-21 20:37:31 +08:00
Wang Yan
22021a988e
Merge pull request #10753 from wy65701436/artifact-trash
add artifact trash manager
2020-02-19 19:14:48 +08:00
Steven Zou
f1374737f6
Merge pull request #10694 from danielpacak/feature/install_with_trivy
chore(install): Add --with-trivy arg to the installation script
2020-02-19 16:27:57 +08:00
Wang Yan
f160505686 add artifact trash manager
1, move the deleted artifact into trash
2, disable GC to delete the untagged manifest

Signed-off-by: wang yan <wangyan@vmware.com>
2020-02-19 14:52:58 +08:00
Tran Huy
2aed96851e Fix typo
Signed-off-by: Tran Huy <Tran.Huy@ericsson.com>
2020-02-18 13:26:59 +01:00
Wenkai Yin(尹文开)
341cb88cba
Merge pull request #10727 from ywk253100/200214_bump_up_legacy_api_version
Bump up legacy api version to v2.0
2020-02-18 10:37:25 +08:00
Wenkai Yin(尹文开)
d7903fcf1b
Merge pull request #10682 from ywk253100/200126_label
Support add/remove label to/from artifact
2020-02-17 11:33:45 +08:00
Daniel Pacak
1b60bb255c refactor(Makefile): Add variables for download URLs
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-02-14 11:58:59 +01:00
Wenkai Yin
94787ea60d Bump up the version of legacy APIs to v2.0
Bump up the version of legacy APIs to v2.0

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-14 13:16:30 +08:00
Daniel Pacak
70dda1387a chore: Configure Redis URL for Trivy adapter
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-02-13 17:57:02 +01:00
Daniel Pacak
4755439b75 chore: Build Trivy adapter from sources
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-02-13 15:32:57 +01:00
Wenkai Yin
93731eeb2e Support add/remove label to/from artifact
This commit add supporting for adding/removing label to/from artifacts and populates labels when listing artifacts

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-02-13 10:46:23 +08:00
Daniel Pacak
a642667ffc chore(install): Add --with-trivy arg to the installation script
Signed-off-by: Daniel Pacak <pacak.daniel@gmail.com>
2020-02-12 23:47:56 +01:00
Ziming Zhang
6047a8102a fix(feat) fix prepare version in installer
Signed-off-by: Ziming Zhang <zziming@vmware.com>
2020-02-06 10:58:29 +08:00
Will Sun
acfcd2d175
Merge pull request #10489 from AllForNothing/postinstall
Fix postinstall script in Docker
2020-02-03 14:13:12 +08:00
Daniel Jiang
2064a1cd6d Switch to basic authentication for registry
1. Add basic authorizer for registry which modify the request
to add basic authorization header to request based on configuration.
2. Set basic auth header for proxy when accessing registry
3. Switche the registry to use basic auth by default and use the basic
authorizer to access Harbor.
4. Make necessary change to test cases, particularly
"test_robot_account.py" and "docker_api.py", because the error is
changed after siwtched to basic auth from token auth.  #10604 is opened
to track the follow up work.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2020-01-31 21:46:47 +09:00
Wenkai Yin
7dc28bcff9 Add foreign key to avoid the concurrent issue
Add foreign key to avoid the concurrent issue when operating the artifacts, tags and references

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2020-01-21 16:03:51 +08:00
sshijun
3175d5f646 Fix postinstall script in Docker
Signed-off-by: sshijun <sshijun@vmware.com>
2020-01-15 16:28:57 +08:00
Daniel Jiang
a087ba02e3 Populate basic auth information for registry
This commit updates `prepare` and templates to populate the credential
for registry for basic authentication.

A temporary flag `registry_use_basic_auth` was added to avoid breakage.
It MUST be removed before the release.

Signed-off-by: Daniel Jiang <jiangd@vmware.com>
2019-12-31 14:50:46 +08:00
Wenkai Yin(尹文开)
5f11ca4df6
Merge pull request #10248 from ywk253100/191213_controller
Define the interface for artifact manager
2019-12-24 17:44:18 +08:00
Wenkai Yin
ac605db5da Define the controller/manager interface for artifact and tag
1. Define the controller/manager interface for artifact and tag
    2. Provide a null implementation for artifact manager

Signed-off-by: Wenkai Yin <yinw@vmware.com>
2019-12-24 17:12:55 +08:00
Ziming
e32649adb4 enhance[cicd] introduce github action for CICD
In order to replace travis.
Implement 5 CI jobs
- UTTEST
- APITEST_DB
- APITEST_LDAP
- OFFLINE
- UI_UT

Signed-off-by: Ziming Zhang <zziming@vmware.com>
2019-12-17 18:36:33 +08:00